Image processing apparatus, image processing method and program

ABSTRACT

An image processing apparatus includes an estimation unit configured to estimate an image generation pixel value corresponding to a position of a phase difference detection pixel of an image element, based on a determination pixel value of image data including the determination pixel value and the image generation pixel value as image data generated by the imaging device including the phase difference detection pixel for generating the determination pixel value for making a focus determination and an image generation pixel for generating the image generation pixel value for generating an image, and an interpolation unit configured to interpolate image generation pixel values of pixels configuring the image data based on the estimated image generation pixel value and the image generation pixel value generated by the image generation pixel.

BACKGROUND

The present disclosure relates to an image processing apparatus and,more particularly, to an image processing apparatus and image processingmethod for interpolating color information and a program causing acomputer to execute the method.

Recently, an imaging apparatus such as a digital camera for imaging anobject such as a person using an imaging device so as to generate animaged image and recording the generated imaged image has been inwidespread use. An imaging device in which color filters are arranged inthe pixels arranged in a light receiving surface in a Bayer array hasbeen in widespread use as the imaging device.

Recently, consequent upon multi-functioning and high resolution ofimaging apparatuses, an imaging device has been considered in whichpixels other than pixels for generating an image are arranged in thepixels arranged in the imaging device or an imaging device in whichpixels utilizing color filters other than the color filters (R, G, B)used for a Bayer array are arranged. For example, an imaging device hasbeen considered in which the existing pixels (image generation pixels)for generating an image and new pixels for multi-function are arrangedin the same imaging device.

For example, an imaging apparatus including such an imaging device, forexample, an imaging apparatus in which pixels (phase differencedetection pixels) for pupil dividing light passing through an imaginglens has been proposed (for example, Japanese Unexamined PatentApplication Publication No. 2009-145401). This imaging apparatus shieldshalf subject light received by a light receiving element and providingphase difference detection pixels for performing pupil division so as toform a pair of images and measures a gap between the formed images so asto calculate a focus deviation. This imaging apparatus calculates amovement quantity of an imaging lens based on the calculated focusdeviation and adjusts the position of the imaging lens based on thecalculated movement quantity, thereby performing focus control.

SUMMARY

In the above-described related art, since both the phase differencedetection pixel and the image generation pixel are included in oneimaging device, it may not be necessary to separately provide twoimaging devices including an imaging device for focus detection and animaging device for an imaged image to the imaging apparatus.

In the above-described related art, if an interpolation process isperformed with respect to image data generated by the imaging device,color information of the phase difference detection pixel and the imagegeneration pixels is interpolated (demosaicing process) using the pixelvalue of the image generation pixel. Accordingly, in comparison with thecase where an interpolation process is performed with respect to imagedata generated by an imaging device which does not include phasedifference detection pixels and includes only image generation pixels,the ratio of pixel values used for the interpolation process isdecreased. In the case where an interpolation process is performed withrespect to image data generated by an imaging device including phasedifference detection pixels and image generation pixels, it is importantto prevent the image quality of the image data from deteriorating due tothe decrease in the ratio of the pixel values used for the interpolationprocess.

The present disclosure has been made in view of such circumstances, itis desirable to improve the image quality of the image data.

According to an embodiment of the present disclosure, there are providedan image processing apparatus including an estimation unit configured toestimate an image generation pixel value corresponding to a position ofa phase difference detection pixel of an image element, based on adetermination pixel value of image data including the determinationpixel value and the image generation pixel value as image data generatedby the imaging device including the phase difference detection pixel forgenerating the determination pixel value for making a focusdetermination and an image generation pixel for generating the imagegeneration pixel value for generating an image, and an interpolationunit configured to interpolate image generation pixel values of pixelsconfiguring the image data based on the estimated image generation pixelvalue and the image generation pixel value generated by the imagegeneration pixel, an image processing method, and a program for causinga computer to execute the method. Accordingly, it is possible to performthe interpolation process of the image generation pixel values using thedetermination pixel value generated by the phase difference detectionpixel.

The image processing apparatus may further include a distanceinformation generation unit configured to generate distance informationbased on a defocus quantity calculated based on the determination pixelvalue for each phase difference detection pixel, and the interpolationunit may interpolate the image generation pixel values of pixelsconfiguring the image data based on the estimated image generation pixelvalue, the distance information and the image generation pixel valuegenerated by the image generation pixel. Accordingly, it is possible toperform the interpolation process of the image generation pixel valuesusing the distance information generated by the distance informationgeneration unit. In this case, the interpolation unit may set a color tobe an interpolation of a reference pixel as a target color using a pixelto be interpolated as the reference pixel, calculate distanceinformation of the image generation pixel based on distance informationof the phase difference detection pixel if the image generation pixelvalue of the reference pixel is interpolated using the distanceinformation, set a pixel which holds an image generation pixel valuerelated to the target color among pixels located in a predeterminedrange from the reference pixel as a target pixel, detect the targetpixel within the predetermined range from the reference pixel in adistance specified by the distance information of the target pixel basedon the distance information, and interpolate the image generation pixelvalues by setting an average value of the image generation pixel valuesrelated to the target color of the detected target pixel to the imagegeneration pixel value related to the target color of the pixel to beinterpolated. Accordingly, it is possible to detect the target pixelclose to the reference pixel in the distance specified by the distanceinformation of the target pixel based on the distance information andinterpolate the image generation pixel values by setting the averagevalue of the image generation pixel values related to the target colorof the detected target pixel to the image generation pixel value relatedto the target color of the pixel to be interpolated. In this case, theinterpolation unit may create a frequency distribution of pixels for thedistance specified by the distance information of the target pixel, seta distance, in which a frequency of the frequency distribution belongsto a maximum class, as a reference, detect the target pixelcorresponding to the distance within the predetermined range from thereference, and set the average value of the image generation pixel valuerelated to the target color of the detected target pixel as the imagegeneration pixel value related to the target color of the pixel to beinterpolated. Accordingly, it is possible to detect the target pixel bythe frequency distribution (histogram).

The phase difference detection pixel may include a microlens forfocusing subject light, a light receiving element for generating animage generation pixel value by receiving the subject light and a lightshielding unit disposed between the microlens and the light receivingelement for partially shielding the subject light, and the estimationunit may calculate the pixel value related to light shielded by thelight shielding unit of the phase difference detection pixel to beestimated based on the determination pixel value generated by the phasedifference detection pixel to be estimated and the determination pixelvalue generated by the phase difference detection pixel adjacent to thephase difference detection pixel to be estimated, and estimate the imagegeneration pixel value of the position of the phase difference detectionpixel based on the calculated pixel value and the determination pixelvalue of the phase difference detection pixel to be estimated.Accordingly, it is possible to calculate the pixel value related tolight shielded by the phase difference detection pixel light shieldingunit and estimate the image generation pixel value of the position ofthe phase difference detection pixel based on the calculated pixel valueand the determination pixel value of the phase difference detectionpixel to be estimated. In this case, the image generation pixel mayinclude a red pixel covered by a red filter for shielding light otherthan a wavelength region indicating red, a blue pixel covered by a bluefilter for shielding light other than a wavelength region indicatingblue, and a green pixel covered by a green filter for shielding lightother than a wavelength region indicating green, the phase differencedetection pixel may be covered by a white filter or a transparent layertransmitting light of a visible light region, and the estimation unitmay estimate the image generation pixel value related to white as theimage generation pixel value of the position of the phase differencedetection pixel. Accordingly, it is possible to estimate the imagegeneration pixel value related to the white as the image generationpixel value of the position of the phase difference detection pixel.

The interpolation unit may set a color to be an interpolation of areference pixel to a target color using a pixel to be interpolated asthe reference pixel, interpolate the image generation pixel valuerelated to white of the image generation pixel values of pixelsconfiguring the image data, and then interpolate the image generationpixel values related to the target color of the reference pixel, basedon the image generation pixel value related to the target color of thepixels located within a predetermined range from the reference pixel,the image generation pixel value related to white of the pixels coveredby the same filter as the pixel to be interpolated as the pixels locatedwithin the predetermined range from the reference pixel and the imagegeneration pixel value related to the white of the reference pixel.Accordingly, it is possible to interpolate the image generation pixelvalues related to the target color of the reference pixel, based on theimage generation pixel value related to the target color of the pixelslocated within the predetermined range from the reference pixel, theimage generation pixel value related to white of the pixels covered bythe same filter as the pixel to be interpolated as the pixels locatedwithin the predetermined range from the reference pixel and the imagegeneration pixel value related to the white of the reference pixel.

The interpolation unit may interpolate the image generation pixel valuesrelated to the target color of the reference pixel, based on a lowfrequency component of the image generation pixel value related to thewhite calculated based on the image generation pixel value related tothe white of the pixels located within the predetermined range from thereference pixel, a low frequency component of the image generation pixelvalue related to the target color calculated based on the imagegeneration pixel value related to the target color of the pixels coveredby the filter of the target color located within the predetermined rangefrom the reference pixel, and the image generation pixel value relatedto the white of the reference pixel. Accordingly, it is possible tointerpolate the image generation pixel values related to the targetcolor of the reference pixel, based on the low frequency component ofthe image generation pixel value related to the target color and theimage generation pixel value related to the white of the referencepixel.

In the imaging device, a first line configured by arranging the imagegeneration pixels in a specific direction and a second line configuredby arranging the phase difference detection pixels in the specificdirection may be alternately arranged in an orthogonal directionorthogonal to the specific direction, and the interpolation unit mayinterpolate the image generation pixel values related to the white ofthe green pixel, based on a low frequency component of the imagegeneration pixel value related to the white calculated from the imagegeneration pixel value related to the white of the phase differencedetection pixels located within the predetermined range in theorthogonal direction of the reference pixel and a low frequencycomponent of the image generation pixel value related to the whitecalculated from the image generation pixel value related to green of thegreen pixels located within the predetermined range, and the imagegeneration pixel value related to green of the reference pixel.Accordingly, it is possible to interpolate the image generation pixelvalues of the image data generated by the imaging device in which thefirst line configured by arranging the image generation pixels in thespecific direction and the second line configured by arranging the phasedifference detection pixels in the specific direction are alternatelyarranged in an orthogonal direction orthogonal to the specificdirection.

According to another embodiment of the present disclosure, there isprovided an image processing apparatus including a distance informationgeneration unit configured to generate distance information based on adefocus quantity calculated for each phase difference detection pixelbased on a determination pixel value of image data including thedetermination pixel value and an image generation pixel value as imagedata generated by an imaging device including a phase differencedetection pixel for generating the determination pixel value for makinga focus determination and an image generation pixel for generating theimage generation pixel value for generating an image; and aninterpolation unit configured to interpolate the image generation pixelvalue of a pixel to be interpolated among pixels configuring the imagedata based on the generated distance information and the imagegeneration pixel value generated by the image generation pixel.Accordingly, it is possible to perform the interpolation process of theimage generation pixel values using the distance information generatedby the distance information generation unit.

According to the embodiments of the present disclosure, it is possibleto obtain excellent effects such as the improvement of the image qualityof image data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing an example of an internalconfiguration of an imaging system according to a first embodiment ofthe present disclosure.

FIG. 2 is a block diagram showing an example of a functionalconfiguration of an imaging apparatus of the imaging system according tothe first embodiment of the present disclosure.

FIG. 3 is a block diagram showing an example of a functionalconfiguration of a demosaicing unit according to the first embodiment ofthe present disclosure.

FIG. 4 is a schematic diagram showing an example of arrangement ofpixels included in the imaging device according to the first embodimentof the present disclosure.

FIGS. 5A and 5B are schematic cross-sectional views showing an exampleof an image generation pixel and an example of a phase differencedetection pixel according to the first embodiment of the presentdisclosure.

FIGS. 6A to 6F are schematic diagrams showing an example of thecalculation of a phase difference pixel W value by a phase differencedetection pixel W value calculation unit according to the firstembodiment of the present disclosure.

FIGS. 7A to 7C are schematic diagrams showing an example of thecalculation of a W value of an R pixel and a B pixel by the R-and-Bpixel W calculation unit according to the first embodiment of thepresent disclosure.

FIGS. 8A and 8B are schematic diagrams showing an example of thecalculation of a W value of an R pixel and a B pixel by the R-and-Bpixel G calculation unit according to the first embodiment of thepresent disclosure.

FIGS. 9A and 9B are schematic diagrams showing an example of ademosaicing process using a W value by the demosaicing unit of theimaging apparatus according to the first embodiment of the presentdisclosure and an example of a demosaicing process of an imagingapparatus of the related art.

FIGS. 10A and 10B are schematic diagrams showing an example of ademosaicing process using distance information by the demosaicing unitof the imaging apparatus according to the first embodiment of thepresent disclosure and an example of a demosaicing process of an imagingapparatus of the related art.

FIG. 11 is a flowchart illustrating an example of an imaging processingprocedure by the imaging apparatus according to the first embodiment ofthe present disclosure.

FIG. 12 is a flowchart illustrating an example of a processing procedureof a demosaicing process (step S930) of an imaging process operationaccording to the first embodiment of the present disclosure.

FIG. 13 is a flowchart illustrating an example of a processing procedureof a phase difference detection pixel W value calculation process (stepS940) by the phase difference detection pixel W value calculation unitaccording to the first embodiment of the present disclosure.

FIG. 14 is a flowchart illustrating an example of a processing procedureof an image generation pixel W value calculation process (step S950) bythe R-and-B pixel W value calculation unit according to the firstembodiment of the present disclosure.

FIG. 15 is a flowchart illustrating an example of a processing procedureof a G value calculation process (step S960) by an R-and-B pixel G valuecalculation unit and a phase difference detection pixel G valuecalculation unit according to the first embodiment of the presentdisclosure.

FIG. 16 is a flowchart illustrating an example of a processing procedureof an R value calculation process (step S970) by an R value calculationunit according to the first embodiment of the present disclosure.

FIG. 17 is a flowchart illustrating an example of a processing procedureof a B value calculation process (step S980) by a B value calculationunit according to the first embodiment of the present disclosure.

FIG. 18 is a block diagram showing an example of a functionalconfiguration of a demosaicing unit according to a second embodiment ofthe present disclosure.

FIGS. 19A and 19B are schematic diagrams showing an example of thecalculation of a W value of a G pixel by a G pixel W value calculationunit according to the second embodiment of the present disclosure.

FIG. 20 is a flowchart illustrating a processing procedure of an imagegeneration pixel W an example of value calculation process (step S991)by a G pixel W value calculation unit and an R-and-B pixel W valuecalculation unit according to the second embodiment of the presentdisclosure.

FIG. 21 is a flowchart illustrating an example of a processing procedureof a G value calculation process (step S995) by an R-and-B pixel G valuecalculation unit and a phase difference detection pixel G valuecalculation unit according to the second embodiment of the presentdisclosure.

FIG. 22 is a schematic diagram showing an example of arrangement ofpixels of an imaging device in which image generation pixels arranged intwo columns and phase difference detection pixels arranged in twocolumns are alternately arranged as a modified example of the firstembodiment of the present disclosure.

FIG. 23 is a schematic diagram showing an example of arrangement ofpixels of an imaging device in which phase difference detection pixelseach including a G filter are arranged as a modified example of thefirst embodiment of the present disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, modes (hereinafter, referred to as embodiments) of thepresent disclosure will be described. A description will be given in thefollowing order.

1. First Embodiment (Imaging Control: Example of Demosaicing Processusing Luminance Value of Phase Difference Detection Pixel)

2. Second embodiment (Imaging Control: Example of Calculating W Value ofG Pixel from Correlation between G Value and W Value)

3. Modified Example

1. First Embodiment Internal Configuration Example of Imaging System

FIG. 1 is a schematic diagram showing an example of an internalconfiguration of an imaging system 10 according to a first embodiment ofthe present disclosure. This imaging system 10 includes an imagingapparatus 100 and an interchangeable lens 170.

In addition, in the first embodiment of the present disclosure, theimaging system 10 is a single lens reflex camera for imaging an image,in which a lens is interchangeable. In FIG. 1, for convenience ofdescription, an internal configuration (for example, a configuration ofa flash) which is little used when an image is imaged will be omitted.

In FIG. 1, for convenience of description, only a configuration relatedto the driving of a focus lens is shown with respect to the driving of alens and a configuration related to the driving of a zoom lens will beomitted.

The imaging apparatus 100 images an object, generates image data(digital data), and records the generated image data as image content(still image content or moving image content). Hereinafter, an exampleof recording still image content (still image file) as image content(image file) is mainly described. This imaging apparatus 100 includes ashutter unit 112, an imaging device 113, an analog front end (AFE) 114,a correction circuit 115, a phase difference computing circuit 150, awhite balance (WB) circuit 116, and a γ correction circuit 117. Theimaging apparatus 100 includes a demosaicing unit 140, an image memory119, a battery 121, a power supply circuit 122, a communicationinterface (I/F) 123, a card I/F 124 and a memory card 125. In addition,the imaging apparatus 100 includes a video random access memory (VRAM)126, a liquid crystal display (LCD) 127, an operation unit 128, ashutter driving control unit 131 and a shutter driving motor (M1) 132.The imaging apparatus 100 includes a diaphragm driving control unit 133,a focus driving control unit 134, a main control unit 136, andconnection terminals 161 to 163.

The shutter unit 112 opens and closes an optical path of incident lightfrom a subject incident to the imaging device 113 by a curtain bodywhich moves in all directions and is driven by the shutter driving motor(M1) 132. The shutter unit 112 supplies the incident light from thesubject to the imaging device 113 if the optical path is opened.

The imaging device 113 photoelectrically converts the incident lightfrom the subject into an electrical signal, receives the incident lightfrom the subject, and generates an analog electrical signal. The imagingdevice 113 is realized by, for example, a Complementary Metal OxideSemiconductor (CMOS) sensor and a Charge Coupled Device (CCD) sensor. Inthe imaging device 113, a pixel (image generation pixel) for generatinga signal for generating an imaged image based on the light received fromthe subject and phase difference detection pixel for generating a signalfor performing phase difference detection.

In the imaging device 113, as the image generation pixel, a pixel (Rpixel) for receiving red light by a color filter for transmitting red(R) light and a pixel (G pixel) for receiving green light by a colorfilter for transmitting green (G) light are arranged. In addition, inthe imaging device 113, in addition to the R pixel and the G pixel, asthe image generation pixel, a pixel (B pixel) for receiving blue lightby a color filter for transmitting blue (B) light is arranged. Theimaging device 113 will be described with reference to FIG. 4. Theimaging device 113 supplies an electrical signal (analog image signal)generated by photoelectric conversion to the AFE 114.

The AFE 114 performs predetermined signal processing with respect to theanalog image signal supplied from the imaging device 113 and performs,for example, signal processing such as noise elimination and signalamplification with respect to the analog image signal. The AFE 114converts the image signal subjected to signal processing into a digitalsignal and generates the digital image signal. The AFE 114 generates atiming pulse for an imaging operation of the imaging device 113 based ona reference clock supplied from the main control unit 136 and suppliesthe generated timing pulse to the imaging device 113. The AFE 114supplies the generated digital image signal (pixel value) to thecorrection circuit 115.

The correction circuit 115 performs predetermined signal processing withrespect to the image signal supplied from the AFE 114 and corrects theimage signal. The correction circuit 115 performs, for example, blacklevel correction, defect correction, shading correction, mixed colorcorrection, or the like. Here, black level correction is a process ofadjusting a black level such that a pixel value becomes “0” if thequantity of received light is “0” by subtracting a pixel value generatedin a pixel of a region which is typically shielded from each pixelvalue. Defect correction is a process of estimating and correcting apixel value of a pixel (defective pixel) which does not normally performa function in the imaging device 113 from a pixel value of a peripheralpixel of the defective pixel. Shading correction is a process ofcorrecting deterioration (shading) in luminance which occurs as a pixelposition is shifted from the center of the imaging device 113 to theperipheral part by applying gain according to an image height to pixelvalues of the overall image. Mixed color correction is a process ofcorrecting increase (mixed color) of a pixel value by light leaked froman adjacent pixel by estimating increment of the mixed color andperforming subtraction. The correction circuit 115 supplies the imagesignal generated by a phase difference detection pixel among the imagesignals subjected to such correction processing to the phase differencecomputing circuit 150. The correction circuit 115 supplies the imagesignal (the image signal generated by the phase difference detectionpixel and the image generation pixel) to the WB circuit 116.

The WB circuit 116 performs a process (so-called white balancecorrection) of correcting color balance of the imaged image based on apredetermined reference color which is white. The WB circuit 116supplies the image signal subjected to white balance correction to the γcorrection circuit 117.

The γ correction circuit 117 corrects a gray scale property of the imagedata subjected to white balance correction. More specifically, the γcorrection circuit 117 performs a non-linear conversion (so-called γcorrection) with respect to the pixel value generated by each pixelusing a predetermined gamma correction table. The γ correction circuit117 supplies the image signal subjected to γ correction to thedemosaicing unit 140.

The phase difference computing circuit 150 detects deviation of focususing a phase difference detection method based on the image signalgenerated by the phase difference detection pixel supplied from thecorrection circuit 115. The phase difference computing circuit 150performs computation for deviation of focus of a focused object, forautofocus (AF), and supplies information about the detected focus to themain control unit 136. The phase difference computing unit 150 detectsdeviation of focus of a pair of phase difference detection pixels andsupplies information about the detected focus to the demosaicing unit140, for each pixel of phase difference detection pixels.

The demosaicing unit 140 performs a demosaicing process (interpolationprocess) such that all channels of R, G and B are aligned at each pixelposition. The demosaicing unit 140 performs a demosaicing process basedon the image signal generated by the phase difference detection pixel,the image signal generated by the image generation pixels (R pixel, Gpixel and B pixel), and information about focus. The demosaicing unit140 supplies the image signal subjected to the demosaicing process tothe image memory 119.

The image memory 119 temporarily holds the image signal supplied fromthe demosaicing unit 140. The image memory 119 is used as a workingregion for performing predetermined processing with respect to the imageaccording to a control signal from the main control unit 136. The imagememory 119 temporarily holds the image signal read from the memory card125.

The battery 121 supplies power for operating the imaging system 10 andincludes, for example, a secondary battery such as a nickel-hydrogenrechargeable battery. The battery 121 supplies power to the power supplycircuit 122.

The power supply circuit 122 converts power supplied from the battery121 into a voltage for operating each unit of the imaging system 10. Forexample, the power supply circuit 122 generates a voltage of 5V andsupplies the generated voltage to the main control unit 136, if the maincontrol unit 136 operates at the voltage of 5V. The power supply circuit122 supplies the generated voltage to each unit of the imaging system10. In FIG. 1, the power supply line from the power supply circuit 122to each unit is partially omitted.

The communication I/F 123 is an interface for enabling data transmissionbetween an external device and the main control unit 136.

The card I/F 124 is an interface for enabling data transmission betweenthe memory card 125 and the main control unit 136.

The memory card 125 is a storage medium for holding an image signal andholds data supplied through the card I/F 124.

The VRAM 126 is a buffer memory for temporarily holding an imagedisplayed on the LCD 127 and supplies the held image to the LCD 127.

The LCD 127 displays an image under the control of the main control unit136 and the LCD 127 includes, for example, a color liquid crystal panel.The LCD 127 displays an imaged image, a recorded image, a mode settingscreen, or the like.

The operation unit 128 receives a user operation and supplies a presssignal to the main control unit 136, for example, when a shutter button(not shown) is pressed. The operation unit 128 supplies a signal relatedto a user operation to the main control unit 136.

The shutter driving control unit 131 generates a driving signal fordriving the shutter driving motor (M1) 132 based on a shutter controlsignal supplied from the main control unit 136 and supplies thegenerated driving signal to the shutter driving motor (M1) 132.

The shutter driving motor (M1) 132 is a motor for driving the shutterunit 112 based on the driving signal supplied from the shutter drivingcontrol unit 131.

The diaphragm driving control unit 133 generates a signal (diaphragmdriving control signal) for controlling the driving of a diaphragm basedon information about a diaphragm supplied from the main control unit 136and supplies the generated diaphragm driving signal to theinterchangeable lens 170 through the connection terminal 161.

The main control unit 136 controls the operation of each unit of theimaging apparatus 100 and includes, for example, a macro computerincluding a ROM for storing a control program.

The focus driving control unit 134 generates a driving quantity signalindicating the driving quantity of a lens based on information aboutfocus supplied from the main control unit 136. The focus driving controlunit 134 supplies the generated driving quantity signal to theinterchangeable lens 170 through the connection terminal 163.

The interchangeable lens 170 includes a plurality of lenses, focuseslight of an image imaged by the imaging apparatus 100 and forms an imageon an imaging surface using the focused light. The interchangeable lens170 includes a diaphragm driving mechanism 181, a diaphragm drivingmotor (M3) 182, a lens position detection unit 183, a lens drivingmechanism 184, a lens driving motor (M4) 185 and a lens barrel 190. Thelens barrel 190 includes a diaphragm 191 and a lens group 194. In thelens group 194, for convenience of description, only a zoom lens 192 anda focus lens 193 are shown.

The diaphragm driving mechanism 181 generates a driving signal fordriving the diaphragm driving motor (M3) 182 based on a diaphragmdriving control signal supplied through the connection terminal 161. Thediaphragm driving mechanism 181 supplies the generated driving signal tothe diaphragm driving motor (M3) 182.

The diaphragm driving motor (M3) 182 is a motor for driving thediaphragm 191 based on the driving signal supplied from the diaphragmdriving mechanism 181. The diaphragm driving motor (M3) 182 changes thediaphragm diameter of the diaphragm 191 by driving the diaphragm 191.

The lens position detection unit 183 detects the position of the zoomlens 192 and the focus lens 193 of the lens group 194. The lens positiondetection unit 183 supplies information (lens position information)about the detected position to the imaging apparatus 100 through theconnection terminal 162.

The lens driving mechanism 184 generates a driving signal for drivingthe lens driving motor (M4) 185 based on a driving quantity signalsupplied through the connection terminal 163. The lens driving mechanism184 supplies the generated driving signal to the lens driving motor (M4)185.

The lens driving motor (M4) 185 is a motor for driving the focus lens193 based on the driving signal supplied from the lens driving mechanism184. The lens driving motor (M4) 185 adjusts focus by driving the focuslens 193.

The lens barrel 190 is a part in which lenses configuring the lens group194 of the interchangeable lens 170 are mounted.

The diaphragm 191 is a shield material for adjusting the quality ofincident light from the subject incident to the imaging apparatus 100.

The zoom lens 192 is moved in the lens barrel 190 in an optical axisdirection to change a focal length and adjust the magnification of thesubject included in the imaged image.

The focus lens 193 is moved in the lens barrel 190 in an optical axisdirection to adjust focus.

Functional Configuration Example of Imaging System

FIG. 2 is a block diagram showing an example of a functionalconfiguration of the imaging apparatus 100 of the imaging system 10according to the first embodiment of the present disclosure.

In the same figure, each related configuration until an image developedafter the imaging device generates an image signal is generated will bedescribed.

The imaging apparatus 100 includes an imaging device 210, an analogsignal processing unit 220, an A/D converter 230, a black levelprocessing unit 240, an image correction unit 250, and a distanceinformation calculation unit 260. The imaging apparatus 100 includes aWB processing unit 270, a γ correction unit 275, a demosaicing unit 300,an image processing unit 280, a display unit 281, and a recording unit282.

The imaging device 210 photoelectrically converts the incident lightfrom the subject into an electrical signal so as to generate an imagesignal. The imaging device 210 supplies an analog image signal generatedby the image generation pixel and the phase difference detection pixelarranged in the imaging device 210 to the analog signal processing unit220. The imaging device 210 corresponds to the imaging device 113 shownin FIG. 1.

The analog signal processing unit 220 performs a predetermined analogsignal process with respect to the analog image signal supplied from theimaging device 210. The analog signal processing unit 220 is realizedby, for example, a correlated double sampling (CDS) circuit, an autogain control (AGC) circuit, a clamp circuit and the like. That is, theanalog signal processing unit 220 performs noise elimination, signalamplification or the like of the analog image signal. The analog signalprocessing unit 220 supplies the analog image signal subjected to signalprocessing to the A/D converter 230. The analog signal processing unit220 corresponds to the AFE 114 shown in FIG. 1.

The A/D converter 230 converts the analog image signal supplied from theanalog signal processing unit 220 into a digital image signal (pixelvalue). That is, the A/D converter 230 changes a signal of a continuousquantity of each pixel to a signal indicated a numerical value. The A/Dconverter 230 supplies the converted digital image signal to the blacklevel processing unit 240. The A/D converter 230 corresponds to the AFE114 shown in FIG. 1.

The black level processing unit 240 performs black level correction withrespect to the pixel value supplied from the A/D converter 230. Theblack level processing unit 240 supplies the image signal subjected toblack level correction to the image correction unit 250. The black levelprocessing unit 240 corresponds to the correction circuit 115 shown inFIG. 1.

The image correction unit 250 performs detection such as defectcorrection, shading correction and mixed color correction with respectto the image signal supplied from the black level processing unit 240.The image correction unit 250 supplies the image signal generated by thephase difference detection pixel of the image signal subjected tocorrection to the distance information calculation unit 260. The imagecorrection unit 250 supplies the image signal subjected to correction(the image signal of the signal of both pixels (the image generationpixel and the phase difference detection pixel) to the WB processingunit 270. The image correction unit 250 corresponds to the correctioncircuit 115 shown in FIG. 1.

The distance information calculation unit 260 calculates distanceinformation which is information about deviation of focus of the place,in which the phase difference detection pixel is arranged, based on theimage signal of the phase difference detection pixel supplied from theimage correction unit 250. The distance information calculation unit 260detects deviation of focus (defocus quantity) of the place where a pairof phase difference detection pixels is arranged, for each pair of thephase difference detection pixel. The distance information calculationunit 260 supplies information about the detected deviation of focus(e.g., a value indicating the quantity of deviation of a pair of imagesby the number of pixels) to the demosaicing unit 300 through the signalline 261 as distance information. The distance information calculationunit 260 detects deviation of focus for a focused object and suppliesthe detected deviation of focus to the main control unit 136 through thesignal line 262. In addition, the distance information calculation unit260 corresponds to a phase difference computing circuit 150 shown inFIG. 1. The distance information calculation unit 260 corresponds to adistance information generation unit described in claims.

The WB processing unit 270 performs white balance correction withrespect to the image signal supplied from the image correction unit 250.The WB processing unit 270, for example, specifies a part estimated asoriginally white from the image signal of the image generation pixel (Rpixel, G pixel and B pixel) so as to calculate correction gain andadjusts (corrects) level balance of the image signal of the phasedifference detection pixel, the R pixel, the G pixel and the B pixel.The WB processing unit 270 supplies the image signal after correction tothe γ correction unit 275. The WB processing unit 270 corresponds to theWB circuit 116 shown in FIG. 1.

The γ correction unit 275 performs γ correction with respect to theimage signal supplied from the WB processing unit 270. The γ correctionunit 275 performs γ correction with respect to the image signal of thephase difference detection pixel, the R pixel, the G pixel and the Bpixel and supplies the image signal (image data) correction gammacorrection to the demosaicing unit 300 through the signal line 290. Theγ correction unit 275 corresponds to the γ correction circuit 117 shownin FIG. 1.

The demosaicing unit 300 performs a demosaicing process (interpolationprocess) such that all channels of R, G and B are aligned at each pixelposition, based on the image signal (image data) supplied from the γcorrection unit 275 and the distance information supplied from thedistance information calculation unit 260. That is, through thedemosaicing process through the demosaicing unit 300, all colors (R, Gand B) of the positions of the pixels are aligned so as to develop theimage. The demosaicing unit 300 supplies the image data (RGB image)subjected to the demosaicing process to the image processing unit 280through the signal line 301. The demosaicing unit 300 will be describedin detail with reference to FIG. 3.

The image processing unit 280 generates display image data viewed to auser and recorded image data recorded on a recording medium based on theRGB image supplied from the demosaicing unit 300. The image processingunit 280 reduces the RGB image according to the resolution of thedisplay unit 281 so as to generate the display image data and suppliesthe generated display image data to the display unit 281. The imageprocessing unit 280 compresses the RGB image according to a recordingmethod recorded in the recording unit 282 so as to generate the recordedimage and supplies the generated recorded image data to the recordingunit 282. The image processing unit 280 corresponds to the main controlunit 136 shown in FIG. 1.

The display unit 281 displays an image based on the display image datasupplied from the image processing unit 280. The display unit 281 isrealized by, for example, a color liquid crystal panel. The display unit281 corresponds to the LCD 127 shown in FIG. 1.

The recording unit 282 records the recorded image data supplied from theimage processing unit 280 as image content (image file). For example, asthe recording unit 282, a removable recording medium (one or a pluralityof recording mediums) such as a disc such as a digital versatile disc(DVD) or a semiconductor memory such as a memory card may be used. Thisrecording medium may be mounted in the imaging apparatus 100 and may bedetached from the imaging apparatus 100. The recording unit 282corresponds to the memory card 125 shown in FIG. 1.

Functional Configuration Example of Demosaicing Unit

FIG. 3 is a block diagram showing an example of a functionalconfiguration of the demosaicing unit 300 according to the firstembodiment of the present disclosure.

The demosaicing unit 300 includes a phase difference detection pixel Wvalue calculation unit 310, a B pixel W value calculation unit 330, a Gvalue calculation unit 340, an R value calculation unit 360, a B valuecalculation unit 370 and an image synthesis unit 380.

The phase difference detection pixel W value calculation unit 310calculates the pixel value (W value) of the white (W) color of eachphase difference detection pixel based on the image signal (determinedpixel value) of each phase difference detection pixel of the image data.The W value indicates the pixel value generated by the image generationpixel (referred to as a white (W) pixel) including a filter transmittingall light of a visible light wavelength region. The W pixel includingsuch a filter is, for example, an image generation pixel which does notinclude a filter (includes a transparent layer instead of a filterlayer), an image generation pixel including a filter (white filter)absorbing light of the other wavelength, or the like.

That is, the phase difference detection pixel W value calculation unit310 estimates the pixel value (W value) of the assumed position in thecase where it is assumed that the W pixel (the image generation pixelreceiving all light of the visible light wavelength region) is arrangedat the position of the phase difference detection pixel from the pixelvalue of the phase difference detection pixel. Then, the pixel value(image generation pixel value) generated by the image generation pixelin the case where it is assumed that the image generation pixelincluding a filter having the same color as the phase differencedetection pixel is arranged at the position of each phase differencedetection pixel is generated. The phase difference detection pixel Wvalue calculation unit 310 supplies the calculated W value (hereinafter,referred to as a phase difference detection pixel W value) of the phasedifference detection pixel to the R-and-B pixel W value calculation unit330. The calculation of the phase difference detection pixel W value bythe phase difference detection pixel W value calculation unit 310 willbe described with reference to FIGS. 6A to 6F. The phase differencedetection pixel W value calculation unit 310 is an example of anestimation unit described in the claims.

The R-and-B pixel W value calculation unit 330 calculates the W value ofthe R pixel and the B pixel of the image generation pixel based on thephase difference detection pixel W value supplied from the phasedifference detection pixel W value calculation unit 310 and the distanceinformation supplied from the distance information calculation unit 260through the signal line 261. That is, the R-and-B pixel W valuecalculation unit 330 estimates the pixel value (W value) in the casewhere it is assumed that the W pixel is arranged at the positions of theR pixel and the B pixel from the phase difference detection pixel Wvalue and the distance information. The R-and-B pixel W valuecalculation unit 330 detects a target pixel close to a reference pixelin a distance designated by the distance information in the phasedifference detection pixels (target pixels) within a predetermined rangecentered on a pixel (reference pixel) to be calculated. The R-and-Bpixel W value calculation unit 330 sets an average value of the W value(pixel value related to a target color) of the detected pixel to the Wvalue of the reference pixel. If the target pixel close to the referencepixel is detected by a histogram (frequency distribution), the targetpixel corresponding to the distance within the predetermined range(within a predetermined threshold) from a reference using the distancein which a frequency belongs to a maximum class as the reference. TheR-and-B pixel W value calculation unit 330 supplies the calculated Wvalue of the R pixel (R pixel W value) and W value of the B pixel (Bpixel W value) to the R-and-B pixel G value calculation unit 341 of theG value calculation unit 340. The calculation of the R pixel W value andthe B pixel W value by the R-and-B pixel W value calculation 330 will bedescribed with reference to FIGS. 7A to 7C.

The G value calculation unit 340 calculates (interpolates) the pixelvalue (G value) related to the G color in the pixel other than the Gpixel. Here, the G value is a value indicating a gray scale of G incolor space expression by RGB and is a pixel value generated by thepixel (G pixel) which receives green light by a color filtertransmitting green (G) light. That is, the G value calculation unit 340estimates the G value of the assumed position in the case where it isassumed that the G pixel is arranged at the positions of the phasedifference detection pixel, the R pixel and the B pixel. The G valuecalculation unit 340 includes an R-and-B pixel G value calculation unit341 and a phase difference detection pixel G value calculation unit 342.

The R-and-B pixel G value calculation unit 341 calculates the G value ofthe R pixel and the G value of the B pixel based on the G value of the Gpixel (G pixel G value) supplied through the G line 294 of the signalline 290 and the R pixel W value and the B pixel W value supplied fromthe R-and-B pixel W value calculation unit 330. The R-and-B pixel Gvalue calculation unit 341 calculates the G value of the R pixel (Rpixel G value) and the G value of the B pixel (B pixel G value) based oncorrelation between the W value and the G value and the W value of thepixel to be calculated. That is, in the R-and-B pixel G valuecalculation unit 341, the W value is used as a pixel value forgenerating the pixel value (image generation pixel value) configuring animage. The R-and-B pixel G value calculation unit 341 supplies thecalculated R pixel G value and B pixel G value to the image synthesisunit 380. The calculation of the R pixel G value and the B pixel G valueby the R-and-B pixel G value calculation unit 341 will be described withreference to FIGS. 8A and 8B.

The phase difference detection pixel G value calculation unit 342calculates the G value of the phase difference detection pixel (phasedifference detection pixel G value) based on the G value of the G pixel(G pixel G value) supplied through the G line 294 of the signal line 290and the distance information supplied through the signal line 261. Thephase difference detection pixel G value calculation unit 342 detectsthe G pixel of a distance position having a high frequency within apredetermined range centered on a pixel to be calculated from thehistogram and sets the average value of the G value of the correspondingG pixel (G pixel G value) to the G value of the phase differencedetection pixel to be calculated. That is, the phase differencedetection pixel G value calculation unit 342 calculates the phasedifference detection pixel G value by the calculation method using thehistogram, similarly to the R-and-B pixel W value calculation unit 330.The phase difference detection pixel G value calculation unit 342supplies the calculated phase difference detection pixel G value to theimage synthesis unit 380.

The R value calculation unit 360 calculates (interpolates) the pixelvalue (R value) related to the R color in the pixel other than the Rpixel. Here, the R value is a value indicating a gray scale of R incolor space expression by RGB and is a pixel value generated by thepixel (R pixel) which receives red light by a color filter transmittingred (R) light. The R value calculation unit 360 estimates the R value ofthe phase difference detection pixel, the R value of the G pixel and theR value of the B pixel based on the R value of the R pixel (R pixel Rvalue) supplied through the R line 293 of the signal line 290 and thedistance information supplied through the signal line 261. The R valuecalculation unit 360 calculates the R value of each pixel by thecalculation method using the histogram, similarly to the R-and-B pixel Wvalue calculation unit 330. The R value calculation unit 360 suppliesthe calculated R values (phase difference detection pixel R value, the Gpixel R value and the B pixel R value) to the image synthesis unit 380.

The B value calculation unit 370 calculates (interpolates) the pixelvalue (B value) related to the B color in the pixel other than the Bpixel. Here, the B value is a value indicating a gray scale of B incolor space expression by RGB and is a pixel value generated by thepixel (B pixel) which receives red light by a color filter transmittingblue (B) light. The B value calculation unit 370 estimates the B valueof the phase difference detection pixel, the B value of the R pixel andthe B value of the G pixel based on the B value of the B pixel (B pixelB value) supplied through the B line 292 of the signal line 290 and thedistance information supplied through the signal line 261. The B valuecalculation unit 370 calculates the B value of each pixel by thecalculation method using the histogram, similarly to the R-and-B pixel Wvalue calculation unit 330. The B value calculation unit 370 suppliesthe calculated B values (phase difference detection pixel B value, the Rpixel B value and the G pixel B value) to the image synthesis unit 380.The R-and-B pixel W value calculation unit 330, the R-and-B pixel Gvalue calculation unit 341, the phase difference detection pixel G valuecalculation unit 342, the R value calculation unit 360 and the B valuecalculation unit 370 are an example of an interpolation unit of theclaims.

The image synthesis unit 380 synthesizes a RGB image. The imagesynthesis unit 380 synthesizes image data of an R component of the RGBimage based on the R pixel R value from the R line 293 and the R values(the phase difference detection pixel R value, the G pixel R value andthe B pixel R value) from the R value calculation unit 360. The imagesynthesis unit 380 synthesizes image data of a G component of the RGBimage based on the G pixel G value from the G line 294 and the G values(the R pixel G value, the B pixel G value and the phase differencedetection pixel G value) from the R-and-B pixel G value calculation unit341 and the phase difference detection pixel G value calculation unit342. The image synthesis unit 380 synthesizes image data of a Bcomponent of the RGB image based on the B pixel B value from the B line292 and the B values (the phase difference detection pixel B value, theR pixel B value, the G pixel B value) from the B value calculation unit370. The image synthesis unit 380 supplies the image, in which RGB isaligned, through the signal line 301.

Arrangement Example of Pixels in Imaging Device

FIG. 4 is a schematic diagram showing an example of arrangement ofpixels included in the imaging device 210 according to the firstembodiment of the present disclosure.

In the same figure, a description will be given using an XY axis inwhich a vertical direction is a Y axis and a horizontal direction is anX axis. In the same figure, a left lower corner is an original point ofthe XY axis, a direction from bottom to top is a + side of the Y axisand a direction from left to right is a + side of the X axis. In thesame figure, a specific direction (a direction corresponding to ahorizontal direction (left-and-right direction) of an imaged image) ofthe imaging device 210 is an X-axis direction and an orthogonaldirection (a direction corresponding to a vertical direction(upper-and-lower direction) of an imaged image) orthogonal to a specificdirection is a Y-axis direction. A signal reading direction of theimaging device 210 is an X-axis direction (read in row units).

In the same figure, for convenience of description, a description willbe given using a region (region 410) of some pixels (16×16 pixels) ofpixels configuring the imaging apparatus 210. In the arrangement of thepixels of the imaging device 210, pixel arrangement shown in the region410 is one unit and pixel arrangement corresponding to this unit (pixelarrangement corresponding to the region 410) is arrangement repeated inthe X-axis direction and the Y-axis direction.

In the same figure, one pixel is denoted by one square. The imagegeneration pixel is denoted by a square including a symbol R, G or Bindicating a color filter therein. That is, the R pixel of the imagegeneration pixel is denoted by an R pixel 411 in the same figure and theB pixel of the image generation pixel is denoted by the B pixel 414 inthe same figure. With respect to the G pixel, the G pixel of a row(line) including the R pixel (R pixel 411) is denoted by a Gr pixel (Grpixel 412) and the pixel of a row (line) including the B pixel (B pixel414) is denoted by a Gb pixel (Gb pixel 413).

The phase difference detection pixel is denoted by a gray square towhich a white ellipse is attached. In addition, the white ellipse of thephase difference detection pixel indicates a side in which incidentlight is not shielded by a light shielding layer and is received by alight receiving element (a side in which an opening is present in alight shielding layer). Now, the phase difference detection pixel (phasedifference detection pixels 415 to 418) shown in the same figure will bedescribed.

The phase difference detection pixel 415 is a phase difference detectionpixel in which a light shielding layer is formed such that subject lightpassing through a right half part of an exit pupil among subject lightincident to a microlens of the phase difference detection pixel 415 isshielded. That is, the phase difference detection pixel 415 shieldslight of the right half part of the light pupil divided into the rightand left (+− side of the X-axis direction) of the exit pupil andreceives the pupil divided light of the left half part.

The phase difference detection pixel 416 is a phase difference detectionpixels in which a light shielding layer is formed such that subjectlight passing through a left half part of an exit pupil among subjectlight incident to a microlens of the phase difference detection pixel416 is shielded. That is, the phase difference detection pixel 416shields light of the left half part of the light pupil divided into theright and left (+− side of the X-axis direction) of the exit pupil andreceives the pupil divided light of the right half part. The phasedifference detection pixel 416 is used as a pair with the phasedifference detection pixel 415 to form a pair of images.

The phase difference detection pixel 417 is a phase difference detectionpixel in which a light shielding layer is formed such that subject lightpassing through an upper half part of an exit pupil among subject lightincident to a microlens of the phase difference detection pixel 417 isshielded. That is, the phase difference detection pixel 417 shieldslight of the upper half part of the light pupil divided into the upperand lower side (+− side of the Y-axis direction) of the exit pupil andreceives the pupil divided light of the lower half part.

The phase difference detection pixel 418 is a phase difference detectionpixels in which a light shielding layer is formed such that subjectlight passing through a lower half part of an exit pupil among subjectlight incident to a microlens of the phase difference detection pixel418 is shielded. That is, the phase difference detection pixel 418shields light of the lower half part of the light pupil divided into theupper and lower side (+− side of the Y-axis direction) of the exit pupiland receives the pupil divided light of the upper half part. The phasedifference detection pixel 418 is used as a pair with the phasedifference detection pixel 417 to form a pair of images.

Now, the pixel arrangement of the imaging device 210 will be described.

In the imaging device 210, a row (line) in which the image generationpixels are arranged and a row (line) in which the phase differencedetection pixel are arranged are alternately arranged. That is, as shownin FIG. 3, the image generation pixel, the phase difference detectionpixel, the image generation pixel, the phase difference detection pixel,. . . are alternately arranged in the y-axis direction. In the imagingdevice 210, in the arrangement of only the image generation pixelsexcept the row in which the phase difference detection pixels arearranged, the row in which the B pixel and the G pixel are arranged andthe row in which the R pixel and the G pixel are arranged arealternately arranged so as to become a Bayer array.

In the imaging device 210, a line in which the phase differencedetection pixel 415 and the phase difference detection pixel 416 arearranged and a line in which the phase difference detection pixel 417and the phase difference detection pixel 418 are arranged arealternately arranged with the row of the image generation pixelsinterposed therebetween. That is, in the phase difference detectionpixels, phase difference detection pixels pupil divided in the samedirection (reading direction (right and left)) or the direction (upperand lower side) orthogonal direction orthogonal to the reading directionare arranged in row units.

Next, the cross-sectional configuration of the image generation pixeland the cross-sectional configuration of the phase difference detectionpixel according to the first embodiment of the present disclosure willbe described with reference to FIGS. 5A and 5B.

Configuration Example of Image Generation Pixel and Phase DifferenceDetection Pixel

FIGS. 5A and 5B are schematic cross-sectional views showing an exampleof an image generation pixel and an example of a phase differencedetection pixel according to the first embodiment of the presentdisclosure.

In FIG. 5A, the cross-sectional configuration of the R pixel (R pixel411) of the image generation pixels according to the first embodiment ofthe present disclosure is schematically shown. A difference among theimage generation pixels of three colors (R pixel, G pixel and B pixel)according to the first embodiment of the present disclosure is only adifference in color filter and thus, in FIG. 5A, only thecross-sectional configuration of the R pixel (R pixel 411) will bedescribed. In FIGS. 5A and 5B, the cross-sectional configuration inwhich the right-and-left direction is an X-axis direction and anupper-and-lower direction is a Z-axis direction is shown.

In FIG. 5A, as the cross-section configuration of the R pixel 411, amicrolens 421, an R filter 422, a wire 423, a wire 424 and a lightreceiving element 425 are shown.

The microlens 421 is a lens for focusing subject light to the lightreceiving element 425.

The wire 423 and the wire 424 are wires for connecting each circuit inthe R pixel 411. In FIG. 5A, in the wire 423 and the wire 424, threewires are arranged in a layer shape in an optical axis. The wire 423 andthe wire 424 are arranged so as not to interrupt light incident to thelight receiving element 425.

The light receiving element 425 (photoelectrically) converts thereceived light into an electrical signal so as to generate theelectrical signal of the intensity according to the quantity of receivedlight. The light receiving element 425 includes a photo diode (PD).

In FIG. 5B, the cross-sectional configuration of the phase differencedetection pixel 416 among the phase difference detection pixelsaccording to the first embodiment of the present disclosure. Adifference in cross-sectional configuration between the phase differencedetection pixels 416 to 418 is only a difference in the arrangementdirection of the wires which become light shielding layers and thus, inFIG. 5B, the cross-sectional configuration of the phase differencedetection pixel 416 will be described.

In FIG. 5B, as the cross-sectional configuration of the phase differencedetection pixel 416, a microlens 421, a light receiving element 425, awire 426 and a wire 427 are shown. The microlens 421 and the lightreceiving element 425 are equal to those of FIG. 5A and thus descriptionwill be omitted.

The wire 426 and the wire 427 are wires for connecting each circuit inthe phase difference detection pixel 416. In FIG. 5B, in the wire 426and the wire 427, three wires are arranged in a layer shape in anoptical axis, similarly to the wire 423 and the wire 424 of FIG. 5A.

In the wire 427, one wire protruding to the vicinity of the center ofthe light receiving element 425 is included. This protrusion covers theright half part of the light receiving element 425 between the lightreceiving element 425 and the microlens 421 and shields subject lightpassing the left half part of the exit pupil. The wire 426 is notarranged on the optical path of subject light from the microlens 421 tothe light receiving element 425 and is arranged on the periphery of theoptical path.

In the phase difference detection pixel 416, the half of the lightreceiving element 425 is covered by the wire 427. Thus, in the phasedifference detection pixel 416, half of the light passing through themicrolens 421 is shielded.

Example of the Calculation of Phase Difference Detection Pixel W Valueby Phase Difference Detection Pixel W Value Calculation Unit

FIGS. 6A to 6F are schematic diagrams showing an example of thecalculation of a phase difference detection pixel W value calculationunit 310 according to the first embodiment of the present disclosure.

FIG. 6A schematically shows an example of the calculation of the W valueof the phase difference detection pixel (phase difference detectionpixel 431) which shields light of the left half part of the light pupildivided into the right and left of the exit pupil and receives pupildivided light of the right half part.

In calculation of the W value of the phase difference detection pixel431, the phase difference detection pixel W value calculation unit 310first calculates a pixel value (a light shielding region pixel value)related to light incident to a region shielded by the light shieldinglayer of the phase difference detection pixel 431 (light which isshielded by the light shielding layer and is not received by the lightreceiving element).

The phase difference detection pixel W value calculation unit 310 sumsthe pixel value of the phase difference detection pixel 431 and thecalculated light shielding region pixel value of the phase differencedetection pixel 431 so as to calculate the W value of the phasedifference detection pixel 431.

The light shielding region pixel value (S_(D431)) of the phasedifference detection pixel 431 is calculated, for example, usingEquation 1. The W value (W_(D431)) of the phase difference detectionpixel 431 is calculated, for example, using Equation 2.S _(D431)=(I _(D431)×2+I _(D432)×1)/3  (1)W _(D431)=(I _(D431) +S _(D431))/2  (2)where, I_(D431) denotes the pixel value of the phase differencedetection pixel 431, the W value of which is to be calculated. Inaddition, I_(D432) denotes the pixel value of the phase differencedetection pixel 432 which is closest to the light shielding region ofthe phase difference detection pixel 431.

In FIG. 6A, the light shielding region pixel value (S_(D431))corresponds to the pixel value related to light received by the region433. The W value (W_(D431)) of the phase difference detection pixel 431corresponds to the pixel value related to the W pixel 434 schematicallyshowing the phase difference detection pixel 431 as the W pixel.

As shown in Equation 1, the light shielding region pixel value(S_(D431)) is calculated based on the pixel value (I_(D431)) of thephase difference detection pixel including the light shielding region tobe calculated and the pixel value (I_(D432)) of the phase differencedetection pixel close to the light shielding region to be calculated. Asshown in Equation 1, the pixel value (I_(D431)) and the pixel value(I_(D432)) are multiplied by a weight coefficient (numerical values “2”and “1” of Equation 1) for giving a weight according to a distancebetween the light shielding region to be calculated and a lightreceiving position (white ellipse position) of each phase differencedetection pixel. By performing computation using the weight coefficient,it is possible to improve calculation precision.

As shown in Equation 2, the W value of the phase difference detectionpixel is calculated by a sum of the pixel value (I_(D431)) of the phasedifference detection pixel including the light shielding region to becalculated and the pixel value (S_(D431)) related to light incident tothe light shielding region to be calculated. The W value is an assumedvalue of the pixel value of the image generation pixel which receivesall light of the visible light region and thus becomes higher than the Gvalue, the R value and the B value. As shown in Equation 2, by dividingthe value obtained by summing the pixel value (I_(D431)) of the phasedifference detection pixel and the light shielding region pixel value(S_(D431)) by “2”, the range of the gray scale of the W value becomesclose to the range of the gray scale of the G value.

FIG. 6B schematically shows an example of the calculation of the W valueof the phase difference detection pixel (phase difference detectionpixel 432) which shields the light of the right half part of the lightpupil divided into the right and left of the exit pupil and receivespupil divided light of the left half part. That is, FIG. 6B shows thecalculation of the W value of the phase difference detection pixelarranged as the pair with the phase difference detection pixel shown inthe calculation method of FIG. 6A.

The light shielding region pixel value (S_(D432)) of the phasedifference detection pixel 432 is calculated, for example, usingEquation 3, similarly to the light shielding region pixel value(S_(D431)) of the phase difference detection pixel 431. The W value(W_(D432)) of the phase value detection pixel 432 is calculated, forexample, using Equation 4, similarly to the W value (W_(D431)) of thephase difference detection pixel 431.S _(D432)=(I _(D432)×2+I _(D431)×1)/3  (3)W _(D432)=(I _(D432) +S _(D432))/2  (4)

In FIG. 6B, the light shielding region pixel value (S_(D432))corresponds to the pixel value related to light received by the region437. The W value (W_(D432)) of the phase difference detection pixel 432corresponds to the pixel value related to the W pixel 438 schematicallyshowing the phase difference detection pixel 432 as the W pixel.

FIG. 6C schematically shows an example of the calculation of the W valueof the phase difference detection pixel (phase difference detectionpixel 441) which shields the light of the upper half part of the lightpupil divided into the upper and lower sides of the exit pupil andreceives pupil divided light of the lower half part.

In calculation of the W value of the phase difference detection pixel441, similarly to calculation of the W value of the phase differencedetection pixels 431 and 432 shown in FIGS. 6A and 6B, first, the lightshielding region pixel value of the phase difference detection pixel 441is calculated. The pixel value of the phase difference detection pixel441 and the calculated light shielding region pixel value are summed tocalculate the W value of the phase difference detection pixel 441.

The light shielding region pixel value (S_(D441)) of the phasedifference detection pixel 441 is calculated, for example, usingEquation 5. The W value (W_(D441)) of the phase difference detectionpixel 441 is calculated, for example, using Equation 6.S _(D441)=(I _(D441)×1+I _(D442)×1)/2  (5)W _(D441)=(I _(D441) +S _(D441))/2  (6)where, I_(D441) denotes the pixel value of the phase differencedetection pixel 441, the W value of which is to be calculated. Inaddition, I_(D442) denotes the pixel value of the phase differencedetection pixel 442 used as the pair with the phase difference detectionpixel 441 between (two right and left) phase difference detection pixelswhich are closest to the light shielding region of the phase differencedetection pixel 441.

In FIG. 6C, the light shielding region pixel value (S_(D441))corresponds to the pixel value related to light received by the region443. The W value (W_(D441)) of the phase difference detection pixel 441corresponds to the pixel value related to the W pixel 444 schematicallyshowing the phase difference detection pixel 441 as the W pixel.

As shown in Equation 5, the light shielding region pixel value(S_(D432)) is calculated based on the pixel value of the phasedifference detection pixel including the light shielding region to becalculated and the pixel value of the phase difference detection pixel(the phase difference detection pixel of the pair) close to the lightshielding region to be calculated, similarly to Equation 1 of FIG. 6A.As shown in Equation 5, since the light receiving positions of the phasedifference detection pixels 441 and 442 are positioned at the samedistance from the light shielding region to be calculated, a weightcoefficient (“1”) having the same value is used.

Equation 6 is equal to Equation 2 of FIG. 6A and thus descriptionthereof will be omitted.

FIG. 6D schematically shows an example of the calculation of the W valueof the phase difference detection pixel (phase difference detectionpixel 442) which shields the light of the lower half part of the lightpupil divided into the upper and lower side of the exit pupil andreceives pupil divided light of the upper half part. That is, FIG. 6Dshows the calculation of the W value of the phase difference detectionpixel arranged as the pair with the phase difference detection pixelshown in the calculation method of FIG. 6C.

The method of calculating the light shielding region pixel value(S_(D442)) and the W value (W_(D442)) of the phase difference detectionpixel 442 is equal to the calculation method of the phase differencedetection pixel 441 of FIG. 6C. The light shielding region pixel value(S_(D442)) of the phase difference detection pixel 442 is calculated,for example, using Equation 7. The W value (W_(D442)) of the phasedifference detection pixel 442 is calculated, for example, usingEquation 8.S _(D442)=(I _(D442)×1+I _(D441)×1)/2  (7)W _(D442)=(I _(D442) +S _(D442))/2  (8)

In FIG. 6D, the light shielding region pixel value (S_(D442))corresponds to the pixel value related to light received by the region447. The W value (W_(D442)) of the phase difference detection pixel 442corresponds to the pixel value related to the W pixel 448 schematicallyshowing the phase difference detection pixel 442 as the W pixel.

The W value of the phase difference detection pixel is calculated bycalculating the pixel value related to light shielded by the lightshielding layer and summing the calculated pixel value and the pixelvalue (determined pixel value) of the phase difference detection pixelto be calculated.

FIG. 6E schematically shows the calculation of the W value of each phasedifference detection pixel based on the pixel value of the phasedifference detection pixels arranged in a row, in the row (row 451) inwhich the phase difference detection pixels pupil dividing the exitpupil to the right and left.

FIG. 6F schematically shows the calculation of the W value of each phasedifference detection pixel based on the pixel value of the phasedifference detection pixels arranged in a row, in the row (row 461) inwhich the phase difference detection pixels pupil dividing the exitpupil to the right and left.

As shown in FIGS. 6E and 6F, since the phase difference detection pixelsare arranged in row units, the phase difference detection pixel W valuecalculation unit 310 calculates the W value of the phase differencedetection pixel in row units.

Example of the Calculation of W Value of R Pixel and B Pixel by R-and-BPixel W Value Calculation Unit

FIGS. 7A to 7C are schematic diagrams showing an example of thecalculation of a W value of an R pixel and a B pixel by the R-and-Bpixel W calculation unit 330 according to the first embodiment of thepresent disclosure.

Since the method of calculating the W value of the R pixel and themethod of calculating the W value of the B pixel are identical, theexample of the R pixel will be described.

FIG. 7A shows 9×9 pixels centered on the R pixel, the W value of whichis to be calculated. In FIG. 7A, since the W value of the phasedifference detection pixel is calculated by the phase differencedetection pixel W value calculation unit 310, the phase differencedetection pixel is denoted by a square (hereinafter, referred to as a Wpixel) having a symbol of W. In FIG. 7A, the R pixel, the W value ofwhich is to be calculated, is denoted by a gray R pixel (R pixel 471).In FIG. 7A, the W pixel is surrounded by a thick line.

FIG. 7B shows a histogram created based on a distance specified bydistance information of the phase difference detection pixel in 9×9pixels centered on the R pixel, the W value of which is to be calculatedwhen the R-and-B pixel W calculation unit 330 calculates the W value ofthe R pixel. FIG. 7B shows an example of classifying a value indicatedby the distance information into 15 classes and totaling the 15 classes.

That is, FIG. 7B shows a histogram in which a horizontal axis is an axisindicating the distance classified into 15 classes, a vertical axis isan axis indicating a frequency, and distance information of a phasedifference detection pixel within 9×9 pixels centered on the R pixel,the W value of which is to be calculated is classified into 15 classes.FIG. 7B shows a class (class 473 having a highest frequency and apredetermined threshold (threshold 472) using the distance of this classas a reference.

FIG. 7C schematically shows the W value of the R pixel 471 calculated bythe R-and-B pixel W calculation unit 330. In FIG. 7C, the W value of theR pixel 471 calculated by the R-and-B pixel W calculation unit 330 isshown in a region (W value 476) in which a point is marked. The R value(R value 475) of the R pixel 471 generated by the imaging device 210 isshown.

Now, the flow of the calculation of the W value of the R pixel by theR-and-B pixel W calculation unit 330 will be described. First, theR-and-B pixel W calculation unit 330 selects the phase differencedetection pixels in order to calculate the W value of the R pixel 471.This selection is performed using the histogram created based on thedistance information of the phase difference detection pixels arrangedwithin a predetermined range (in the same figure, an example of 9×9pixels is shown) centered on the R pixel, the W value of which is to becalculated. In FIG. 7A, the phase difference detection pixels arrangedin 9×9 pixels are shown as W pixels surrounded by a thick line. In FIG.7B, the histogram created based on the distance information of the phasedifference detection pixels is shown.

The R-and-B pixel W calculation unit 330 determines a class (473) havinga highest frequency and a class belonging to a distance within apredetermined threshold (threshold 472) using the distance of this classas a reference. In FIG. 7B, the class (473) having the highest frequencyand the class belonging to the distance within the predeterminedthreshold (threshold 472) using the distance of this class as areference is shown by a gray class.

Subsequently, the R-and-B pixel W calculation unit 330 calculates anaverage value of the W values of the phase difference detection pixelsbelonging to the determined class (the gray class of FIG. 7B) and setsthe average value to the W value of the R pixel 471. In FIG. 7C, thecalculated W value of the R pixel 471 is shown as the W value 476.

In this way, the R-and-B pixel W calculation unit 330 detects the phasedifference detection pixel close to the R pixel to be calculated in thedistance (the distance between the imaged matter and the imagingapparatus) based on the distance information of the phase differencedetection pixels arranged within the predetermined range centered on theR pixel to be calculated. The average value of the W values of thedetected phase difference detection pixels becomes the W value of the Rpixel to be calculated. That is, by identifying the imaged matter usingthe distance information, the R-and-B pixel W calculation unit 330improves precision of interpolation of the W value of the R pixel to becalculated.

The threshold 472 is determined by a difference between a maximum valueand a minimum value of the value of the distance information of thephase difference detection pixels. For example, if this difference islarge, the threshold is increased, and, if this difference is small, thethreshold is decreased. In addition, the number of classes and thedistance gap between the classes when the histogram is created isequally determined. The threshold, the number of classes and thedistance gap between the classes may be determined from the distanceinformation of the pixels within the predetermined range centered on thepixel, the W value of which is to be calculated.

The histogram is an example of a method of detecting the phasedifference detection pixels close to the R pixel to be calculated indistance specified by the distance information and the presentdisclosure is not limited thereto. For example, the distance informationof the R pixel to be calculated may be calculated from the distanceinformation of the phase difference detection pixels adjacent to the Rpixel and the phase difference detection pixel including distanceinformation close to the calculated distance information may bedetected.

Example of the Calculation of G Value of R Pixel and B Pixel by R-and-BPixel G Calculation Unit

FIGS. 8A and 8B are schematic diagrams showing an example of thecalculation of a W value of an R pixel and a B pixel by the R-and-Bpixel G calculation unit 341 according to the first embodiment of thepresent disclosure.

Since the method of calculating the G value of the R pixel and themethod of calculating the G value of the B pixel are identical, only theexample of the B pixel will now be described.

FIG. 8A schematically shows 9 pixels in a row in which the B pixel andthe G pixel are arranged. FIG. 8A shows 9 pixels adjacent to the B pixel(B pixel 650), the G value of which is to be calculated, in the rowdirection. The 9 pixels (the B pixels 610, 630, 650, 670 and 690 and thephase difference detection pixels 620, 640, 660 and 680) are pixelswhich hold the pixel values used when calculating the W value of the Bpixel 650.

FIG. 8B shows an example of the calculation of the G value of the Bpixel 650 by the R-and-B pixel G calculation unit 341 along with thepixel values of the 9 pixels shown in FIG. 8A.

As shown in the inside of the 9 pixels arranged in one row of FIG. 8B,when the R-and-B pixel G calculation unit 341 calculates the G value,the pixel value of the B pixel is configured by the B value generated bythe imaging device 210 and the W value calculated by the R-and-B pixel Wvalue calculation unit 330. The pixel value of the G (Gb) pixel isconfigured by the G value generated by the imaging device 210 and the Wvalue calculated by the R-and-B pixel W value calculation unit 330.

In FIG. 8B, the B values of the B pixels (610, 630, 650, 670 and 690)are denoted by B values 613, 633, 653, 573 and 693 and the W values ofthe B pixels are denoted by the W values 614, 634, 654, 674 and 694.

In FIG. 8B, the G values of the Gb pixels (620, 640, 660 and 680) aredenoted by G values 621, 641, 661 and 681 and the W values of the Gbpixels are denoted by the W values 624, 644, 664 and 684.

Now, calculation of the G value of the B pixel 650 by the R-and-B pixelG calculation unit 341 will be described. In calculation of the G valueof the B pixel 650, the R-and-B pixel G calculation unit 341 calculatesthe G value based on high correlation between the G value and the Wvalue and the W value of the B pixel 650.

The G value (G_(B650)) of the B pixel 650 is calculated, for example,using Equations 9 to 11.G _(B650)=(G _(m) /W _(m))×W _(B650)  (9)G _(m) =a×G _(G620) +b×G _(G640) +c×G _(G660) +d×G _(G680)  (10)W _(m) =e×W _(B610) +f×W _(B650) +g×W _(B690)  (11)where, G_(G620) is the G value of the G pixel 620. In addition, G_(G640)is the G value of the G pixel 640, G_(G660) is the G value of the Gpixel 660, G_(G660) is the G value of the G pixel 680. In addition,W_(B610) is the W value of the B pixel 610, W_(B650) is the W value of Bpixel 650, and W_(B690) is the W value of the B pixel 690.

In addition, a to g are weight coefficients. The weight coefficients ato g are set to satisfy the relationships of Equations 12 to 14.a+b+c+d=e+f+g  (12)a=d<b=c  (13)e=h<f  (14)The weight coefficients a to g are set, for example, according to therelationship between the position of the pixel, the G value of which isto be calculated, and the position of the pixel which holds each pixelvalue. That is, G_(m) is a low frequency component of the G valuecalculated by applying a low pass filter using the position of the Bpixel 650 as a reference. In addition, W_(m) is a low frequencycomponent of the W value calculated by applying a low pass filter usingthe position of the B pixel 650 as a reference.

Here, it is assumed that the value of the weight coefficients a and d is“1” and the value of the weight coefficients b and c is “3”, the valueof the weight coefficients e and h is “1”, and the value of the weightcoefficient f is “6”. In this case, Equations 10 and 11 become Equations15 and 16.G _(m)=1×G _(G620)+3×G _(G640)+3×G _(G660)+1×G _(G680)  (15)W _(m)=1×W _(B610)+6×W _(B650)+1×W _(B690)  (16)

G_(m)/W_(m) of the left side of Equation 9 indicates correlation betweenthe low frequency component of the G value using the position of the Bpixel 650 as a reference and the low frequency component of the W valueusing the position of the B pixel 650 as a reference. That is, as shownin Equation 9, the G value of the B pixel 650 is calculated based on thecorrelation between the low frequency components and the W value of theB pixel 650.

In FIG. 8B, the G value (G_(B650)) of the B pixel 650 corresponds to theG value 651 (region in which a point is marked) shown in the B pixel 650shown below an arrow. The pixel values used when calculating the G value(G_(B650)) correspond to the pixel values (the W values 614, 654 and 694and the G values 621, 641, 661 and 681) surrounded by a thick line ofthe 9 pixels arranged in one row.

Calculation of G_(m) and W_(m) is only exemplary and other methods ofsetting various weight coefficients (a method of applying a low passfilter) may be used.

The method of calculating the W value of the R pixel and the B pixel isnot limited thereto and, for example, as shown in FIGS. 7A to 7C, acalculation method using a histogram or the like may be used.

Now, interpolation of the phase difference detection pixel G valuecalculation unit 342, the R value calculation unit 360 and the B valuecalculation unit 370 will be briefly described.

The phase difference detection pixel G value calculation unit 342, the Rvalue calculation unit 360 and the B value calculation unit 370 performinterpolation, similarly to calculation of the W value of the R pixeland the B pixel by the R-and-B pixel W calculation unit 330 shown inFIGS. 7A to 7C.

That is, the distance information of the image generation pixel iscalculated from the distance information of the phase differencedetection pixel adjacent to the image generation pixel. A histogram iscreated based on the calculated distance information, a pixel, the pixelvalue of which is referred to, is determined from the histogram, and anaverage value of the pixel value of a target color of interpolation ofthe referred pixel is calculated, thereby performing interpolation.

Example of Demosaicing Process Using W Value

FIGS. 9A and 9B are schematic diagrams showing an example of ademosaicing process using a W value by the demosaicing unit 300 of theimaging apparatus 100 according to the first embodiment of the presentdisclosure and an example of a demosaicing process of an imagingapparatus of the related art. In FIGS. 9A and 9B, for convenience ofdescription, a region of 6×6 pixels is shown and described.

FIG. 9A shows an example of a demosaicing process of the imagingapparatus of the related art. In FIG. 9A, for convenience ofdescription, it is assumed that arrangement of pixels in the imagingdevice is equal to the arrangement of pixels in the imaging device ofthe embodiment of the present disclosure. FIG. 9A shows a region (region717) indicating the arrangement of pixels in the imaging device, aregion (region 718) indicating the pixel values used in the demosaicingprocess, and a region (region 719) indicating the result of thedemosaicing process.

In the imaging apparatus of the related art including the imaging device(region 717 of FIG. 9A) having the same arrangement of pixels as theembodiment of the present disclosure, the demosaicing process isperformed only using the pixel values generated by the image generationpixels. The pixel values generated by the phase difference detectionpixels are only used to detect a phase difference.

That is, as shown by the region 718, in the demosaicing process, thecolor information of the positions of the image generation pixels andthe phase difference detection pixels are interpolated (demosaicingprocess) based on the pixel value generated by the image generationpixel (the pixel values of 18 pixels surrounded by a thick line of theregion 718). As shown by the region 719, the color information (R value,G value and B value) of all pixels are interpolated even when thedemosaicing process is performed only using the pixel value generated bythe image generation pixel.

However, since only the pixel values generated by the image generationpixels are used as shown by the region 718, the number of pieces ofinformation (pixel values) used for interpolation is decreased by thenumber of phase difference detection pixels. Accordingly, if the imagedata of the imaging device (imaging device 210) of the embodiment of thepresent disclosure in which the number of phase difference detectionpixels becomes equal to the number of image generation pixels issubjected to the demosaicing process, the number of pieces ofinformation used for interpolation is decreased and thus deteriorationin precision of the demosaicing process occurs.

FIG. 9B shows an example of the demosaicing process in the imagingapparatus 100 according to the first embodiment of the presentdisclosure. FIG. 9B shows a region (region 711) indicating thearrangement of pixels in the imaging device 210, a region (region 712)indicating the pixel values used for the demosaicing process and aregion (region 713) indicating the result of the demosaicing process. Inthe region 712, the pixel value of the phase difference detection pixelis denoted by the W value calculated by the phase difference detectionpixel W value calculation unit 310.

In the imaging apparatus 100 according to the first embodiment of thepresent disclosure, the pixel values of the phase difference detectionpixels and the W values of the phase difference detection pixels arecalculated and the demosaicing process is performed using the calculatedW values of the phase difference detection pixels and the pixel valuesgenerated by the image generation pixels. That is, as shown by theregion 712, in the demosaicing process, the color information of thepositions of the image generation pixels and the phase differencedetection pixels are interpolated based on the pixel values of 36 pixelssurrounded by a thick line in the region 712. In the region 713, thecolor information (R value, G value and B value) of all pixels areinterpolated.

In the imaging apparatus 100 according to the first embodiment of thepresent disclosure, the W values of the phase difference detectionpixels are calculated from the pixel values of the phase differencedetection pixels and the W values are used for the demosaicing process.Accordingly, the number of pieces of information used for interpolationis increased and thus precision of the demosaicing process may beimproved.

Example of Demosaicing Process Using Distance Information

FIGS. 10A and 10B are schematic diagrams showing an example of ademosaicing process using distance information by the demosaicing unit300 of the imaging apparatus 100 according to the first embodiment ofthe present disclosure and an example of a demosaicing process of animaging apparatus of the related art. In the imaging apparatus of therelated art, the same apparatus as that shown in FIG. 9A is described.

FIG. 10A shows an imaged image (imaged image 729) after the demosaicingprocess by the imaging apparatus of the related art. In the imaged image729, two buildings having different distances from the imaging apparatusis imaged such that portions thereof overlap each other are shown. It isassumed that the outer walls of the two buildings have similar colors.In the imaged image 729, collapse of the edge between the two buildingsis expressed by representing the edge between the two buildings by adotted line (dotted line 728).

In the demosaicing process of the imaging apparatus of the related art,the color information is interpolated based on the pixel values of thepixels arranged in a predetermined range. That is, a determination as towhether the pixel, the pixel value of which is referred to, is a pixelby which an object different from an object imaged by the pixel to beinterpolated is imaged is not made. Therefore, in the pixels (pixelsadjacent to the edge between the two buildings of the imaged image 729)adjacent to a place where objects having the same color and havingdifferent distances from the imaging apparatus are adjacent to eachother, it is erroneously determined that the adjacent objects are thesame object.

Accordingly, in the demosaicing process of the imaging apparatus of therelated art, interpolation precision may deteriorate.

FIG. 10B shows an imaged image (imaged image 721 after the demosaicingprocess by the imaging apparatus 100 according to the first embodimentof the present disclosure. It is assumed that the subjects of FIG. 10Bare equal to the subjects (two buildings) of FIG. 10A. In the imagedimage 721, the sharpness of the edge between the two buildings isexpressed by representing the edge between the two buildings by a solidline (line 722).

In the demosaicing process according to the first embodiment of thepresent disclosure, the pixel, the pixel value of which is referred to,is selected using the distance information. That is, a determination asto whether the pixel, the pixel value of which is referred to, is apixel by which an object different from an object imaged by the pixel tobe interpolated is imaged is made. The pixel values of the pixels to beinterpolated are interpolated by referring to the pixel values of thepixels by which the same object is imaged.

That is, in the pixels of a place where objects having the same colorand having different distances from the imaging apparatus are adjacentto each other (the pixels adjacent to the edge between the two buildingsof the imaged image 721), the objects imaged by the pixels areidentified using the distance information. Interpolation of the pixelvalues is performed by referring to the pixel values of the pixels bywhich the same object as the pixels to be interpolated is imaged.Accordingly, the edge (line 722) of a place where the objects having thesame color are adjacent to each other becomes sharp (interpolationprecision is improved).

In the imaging apparatus 100 according to the first embodiment of thepresent disclosure, in the demosaicing process, it is possible toimprove prevision of the demosaicing process by selecting the pixels,the pixel values of which are referred to, using the distanceinformation.

Example of Operation of Imaging Apparatus

Next, the operation of the imaging apparatus 100 according to the firstembodiment of the present disclosure will be described with reference tothe figures.

FIG. 11 is a flowchart illustrating an imaging processing procedure bythe imaging apparatus 100 according to the first embodiment of thepresent disclosure. FIG. 11 shows a flow in which the imaging device 210images a subject and the image imaged by imaging is held in therecording unit 282.

First, an imaging process of imaging the subject using the imagingdevice 210 is performed (step S911). Then, through the analog signalprocessing unit 220, an analog signal process of performing apredetermined analog signal process with respect to the analog imagesignal generated by imaging of the imaging device 210 is performed (stepS912).

Thereafter, through the A/D converter 230, the analog image signal afterthe analog signal process of the analog signal processing unit 220 isconverted into a digital image signal by an A/D conversion process (stepS913).

Subsequently, through the black level processing unit 240, a black levelcorrection process of correcting the black level of the digital imagesignal obtained by the A/D converter 230 is performed (step S914).

Through the image correction unit 250, an image correction process ofperforming correction such as defect correction, shading correction,mixed color correction or the like is performed with respect to theimage signal after black level correction (step S915). Thereafter,through the distance information calculation unit 260, a distanceinformation calculation process of calculating the distance informationof a place where the phase difference detection pixels are arranged isperformed based on the image signal of the phase difference detectionpixels supplied from the image correction unit 250 (step S916).

Subsequently, through the WB processing unit 270, a WB process ofcorrecting white balance (WB) of the image signal supplied from theimage correction unit 250 is performed (step S917). Then, through the γcorrection unit 275, the γ correction process is performed with respectto the image signal after the WB process (step S918). Thereafter,through the demosaicing unit 300, a demosaicing process is performedbased on the image signal (image data) after γ correction and thedistance information such that all channels of R, G and B are aligned ateach pixel position (step S930). The demosaicing process (step S930)will be described with reference to FIG. 12.

Through the display unit 281, the imaged image after the demosaicingprocess is displayed (step S919). Through the recording unit 282, theimaged image after the demosaicing process is recorded (step s921). Theimaging process procedure is finished.

FIG. 12 is a flowchart illustrating a processing procedure of ademosaicing process (step S930) of an imaging process operationaccording to the first embodiment of the present disclosure.

First, through the phase difference detection pixel W value calculationunit 310, a phase difference detection pixel W value calculation processof calculating the W value (phase difference detection pixel W value) ofthe phase difference detection pixel is performed (step S940). The phasedifference detection pixel W value calculation process (step S940) willbe described with reference to FIG. 13. Step S940 is an example of anestimation procedure described in the claims.

Subsequently, through the R-and-B pixel W value calculation unit 330, animage generation pixel W value calculation process of calculating the Wvalues of the R pixel and the B pixel among the image generation pixelsis performed based on the phase difference detection pixel W value andthe distance information supplied from the distance informationcalculation unit 260 (step S950). The image generation pixel W valuecalculation process (step S950) will be described with reference to FIG.14.

Through the R-and-B pixel G value calculation unit 341 and the phasedifference detection pixel G value calculation unit 342, a G valuecalculation process of calculating the phase difference detection pixel,the R pixel and the B pixel is performed (step S960). The G valuecalculation process (step S960) will be described with reference to FIG.15.

Thereafter, through the R value calculation unit 360, an R valuecalculation process of calculating the R values of the phase differencedetection pixel, the G pixel and the B pixel is performed (step S970).The R value calculation process (step S970) will be described withreference to FIG. 16.

Subsequently, through the B value calculation unit 370, a B valuecalculation process of calculating the B values of the phase differencedetection pixel, the R pixel and the G pixel is performed (step S980).The B value calculation process (step S980) will be described withreference to FIG. 17. Steps S950, S960, S970 and S980 are an example ofan interpolation procedure of the claims.

Through the image synthesis unit 380, a RGB image is synthesized, thesynthesized image is output (step S931), and the demosaicing processingprocedure is finished.

FIG. 13 is a flowchart illustrating a processing procedure of a phasedifference detection pixel W value calculation process (step S940) bythe phase difference detection pixel W value calculation unit 310according to the first embodiment of the present disclosure.

First, the position of the pixel (pixel to be determined) subjected to adetermination as to whether or not the W value thereof is calculated isset to the position of a start column of a start row of determination(step S941). Subsequently, it is determined whether or not the row (rowto be determined) in which the pixel to be determined is arranged is arow in which a phase difference detection pixel is arranged (step S942).

If it is determined that the row to be determined is the row in whichthe phase difference detection pixel is arranged (step S942), it isdetermined whether or not the pixel to be determined is a phasedifference detection pixel (step S943). If it is determined that thepixel to be determined is not the phase difference detection pixel (stepS943), the process progresses to step S946.

If it is determined that the pixel to be determined is the phasedifference detection pixel (step S943), a pixel value (light shieldingregion pixel value) of a light shielding region included in the phasedifference detection pixel is calculated (step S944). The lightshielding region pixel value is calculated, for example, using Equations1, 3, and 7 shown in FIGS. 6A to 6F. Subsequently, the W value of thephase difference detection pixel is calculated (step S945). The W valueof the phase difference detection pixel is calculated, for example,using Equations 2, 4, 6 and 8 shown in FIGS. 6A to 6F.

Thereafter, it is determined whether or not the position of the pixel tobe determined is a last column (final column) to be determined (stepS946). If it is determined that the position of the pixel to bedetermined is the final column (step S946), the process progresses tostep S948.

If it is determined that the position of the pixel to be determined isnot the final column (step S946), the position of the pixel to bedetermined is shifted by one column (step S947) and the process returnsto step S943.

If it is determined that the row to be determined is not the row inwhich the phase difference detection pixel is arranged (the row of theimage generation pixel) (step S942), it is determined whether or not therow to be determined is a last row (final row) to be determined (stepS948). If it is determined that the row to be determined is not thefinal row (step S948), the position of the pixel to be determined is setto the position of the start column of a row after the row to bedetermined is shifted by one row (step S949) and the process returns tostep S942 to continue to make the determination.

If it is determined that the row to be determined is the final row (stepS948), the phase difference detection pixel W value calculationprocessing procedure is finished.

FIG. 14 is a flowchart illustrating a processing procedure of an imagegeneration pixel W value calculation process (step S950) by the R-and-Bpixel W value calculation unit 330 according to the first embodiment ofthe present disclosure.

First, the position of the pixel (pixel to be determined) subjected to adetermination as to whether or not the W value thereof is calculated isset to the position of a start row of a start column of determination(step S951). Subsequently, it is determined whether or not the pixel tobe determined is a phase difference detection pixel (step S952). If itis determined that the pixel to be determined is the phase differencedetection pixel (step S952), the process progresses to step S956.

If it is determined that the pixel to be determined is not the phasedifference detection pixel (it is image generation pixel) (step S952),it is determined whether or not the pixel to be determined is a G pixel(step S953). If it is determined that the pixel to be determined is theG pixel (step S953), the process progresses to step S956.

If it is determined that the pixel to be determined is not the G pixel(it is an R pixel or a B pixel) (step S953), the W value of the pixel tobe determined (R pixel and B pixel) is calculated based on a histogramusing distance information (step S955). The W value of the R pixel orthe B pixel is calculated, for example, as described with reference toFIGS. 7A to 7C.

Subsequently, it is determined whether or not the position of the pixelto be determined is a last row (final row) (step S956). If it isdetermined that the position of the pixel to be determined is not thefinal row (step S956), the position of the pixel to be determined isshifted by one row (step S957) and the process returns to step S952.

If it is determined that the position of the pixel to be determined isthe final row (step S956), it is determined whether or not the column(column to be determined) in which the pixel to be determined isarranged is a last column (final column) to be determined (step S958).If it is determined that the column to be determined is not the finalcolumn (step S958), the position of the pixel to be determined is set tothe position of the start row of the column after the column to bedetermined is shifted by one column (step S959) and the process returnsto step S952 to continues to make the determination.

If it is determined that the column to be determined is the final column(step S958), the image generation pixel W value calculation processingprocedure is finished.

FIG. 15 is a flowchart illustrating a processing procedure of a G valuecalculation process (step S960) by the R-and-B pixel G value calculationunit 341 and the phase difference detection pixel G value calculationunit 342 according to the first embodiment of the present disclosure.

First, the position of the pixel (pixel to be determined) subjected to adetermination as to whether or not the G value thereof is calculated isset to the position of a start column of a start row of determination(step S961). Subsequently, it is determined whether or not the pixel tobe determined is a G pixel (step S962). If it is determined that thepixel to be determined is the G pixel (step S962), the processprogresses to step S966.

If it is determined that the pixel to be determined is not the G pixel(step S962), it is determined whether or not the pixel to be determinedis a phase difference detection pixel (step S963). If it is determinedthat the pixel to be determined is the phase difference detection pixel(step S963), the distance information of the image generation pixel isset and then the G value of the pixel to be determined is calculatedusing the histogram using the distance information (step S964). Afterthe G value is calculated (step S964), the process progresses to stepS966.

If it is determined that the pixel to be determined is not the phasedifference detection pixel (it is the R pixel or the B pixel) (stepS963), the G value is calculated using correlation between the W valueand the G value (step S965). The G value of the R pixel or the B pixelis calculated, for example, using Equations 9 to 14 shown in FIGS. 8Aand 8B.

Subsequently, it is determined whether or not the position of the pixelto be determined is a last column (final column) (step S966). If it isdetermined that the position of the pixel to be determined is not thefinal column (step S966), the position of the pixel to be determined isshifted by one column (step S967) and then the process returns to stepS962.

If it is determined that the position of the pixel to be determined isthe final column (step S966), it is determined whether or not the row(row to be determined) in which the pixel to be determined pixel isarranged is a last row (final row) to be determined (step S968). If itis determined that the row to be determined is not the final row (stepS968), the position of the pixel to be determined is set to the positionof the start column of the row after the row to be determined is shiftedby one row (step S969) and the process returns to step S962 to continueto the determination.

If it is determined that the row to be determined is the final row (stepS968), the G value calculation processing procedure is finished.

FIG. 16 is a flowchart illustrating a processing procedure of an R valuecalculation process (step S970) by the R value calculation unit 360according to the first embodiment of the present disclosure.

The step S971, step S976 to step S979 of the processing procedurerespectively correspond to step S961, step S966 to step S969 shown inFIG. 15 and thus description thereof will be omitted.

If the position of the pixel to be determined of the R value is set tothe position of the start column of the start row (step S971), it isdetermined whether or not the pixel to be determined is an R pixel (stepS972). If it is determined that the pixel to be determined is the Rpixel (step S972), the process progresses to step S976.

If it is determined that the pixel to be determined is not the R pixel(step S972), the distance information of the image generation pixel isset and the R value of the pixel to be determined (the phase differencedetection pixel, the G pixel or the B pixel) is calculated using thehistogram using the distance information (step S973). After the R valueis calculated (step S973), the process progresses to step S976. The Rvalue is calculated, for example, as described with reference to FIGS.7A to 7C.

FIG. 17 is a flowchart illustrating a processing procedure of a B valuecalculation process (step S980) by the B value calculation unit 370according to the first embodiment of the present disclosure.

Step S981, step S986 to step S989 of the processing procedurerespectively correspond to step S961, step s966 to step S969 shown inFIG. 15 and thus description thereof will be omitted.

If the position of the pixel to be determined of the B value is set tothe position of the start column of the start row (step S981), it isdetermined whether or not the pixel to be determined pixel is a B pixel(step S982). If it is determined that the pixel to be determined is theB pixel (step S982), the process progresses to step S986.

If it is determined that the pixel to be determined is not the B pixel(step S982), the distance information of the image generation pixel isset and then the B value of the pixel to be determined (the phasedifference detection pixel, the R pixel or the G pixel) is calculatedusing the histogram using the distance information (step S983). Afterthe B value is calculated (step S983), the process progresses to stepS986. The B value is calculated, for example, as described withreference to FIGS. 7A to 7C.

According to the first embodiment of the present disclosure, it ispossible to improve image quality of image data by using the pixel valueof the phase difference detection pixel and the distance informationgenerated from the pixel value of the phase difference detection pixelin the demosaicing process.

2. Second Embodiment

Although the example of calculating the G value of the phase differencedetection pixel using the histogram for the distance is described in thefirst embodiment, the present disclosure is not limited thereto andcalculation may be performed based on correlation between the W valueand the G value. Even in the R value and the B value, calculation may beperformed based on correlation between the W value and the R value orcorrelation between the W value and the B value, precision of which islower than that of correlation between the W value and the G value(correlation between the W value and the G value is highest).

In the second embodiment of the present disclosure, an example ofcalculating the W value of the G pixel and calculating the G value ofthe phase difference detection pixel using the calculated W value of theG pixel will be described with reference to FIGS. 18 to 21.

Functional Configuration Example of Demosaicing Unit

FIG. 18 is a block diagram showing an example of a functionalconfiguration of a demosaicing unit 800 according to a second embodimentof the present disclosure.

The configuration of the imaging apparatus according to the secondembodiment of the present disclosure is equal to that of the imagingapparatus 100 except that the demosaicing unit 800 is mounted instead ofthe demosaicing unit 300 of the imaging apparatus 100 according to thefirst embodiment of the present disclosure shown in FIG. 2. Thedescription of the configuration of the imaging apparatus according tothe second embodiment of the present disclosure will now be omitted.

The demosaicing unit 800 includes a phase difference detection pixel Wvalue calculation unit 310, a G pixel W value calculation unit 820, anR-and-B pixel W value calculation unit 330, an R-and-B pixel G valuecalculation unit 341 and a phase difference detection pixel G valuecalculation unit 842. The demosaicing unit 300 includes the R valuecalculation unit 360, the B value calculation unit 370 and the imagesynthesis unit 380. The configuration of the demosaicing unit 800 isequal to that of the demosaicing unit 300 except that the phasedifference detection pixel G value calculation unit 842 is includedinstead of the phase difference detection pixel G value 342 of thedemosaicing unit 300 shown in FIG. 3 and the G pixel W value calculationunit 820 is newly included. Therefore, only the G pixel W valuecalculation unit 820 and the phase difference detection pixel G valuecalculation unit 842 will be described.

The G pixel W value calculation unit 820 calculates the W value of the Gpixel (G pixel W value) based on the phase difference detection pixel Wvalue supplied from the phase difference detection pixel W valuecalculation unit 310 and the G value of the G pixel (G pixel G value)supplied through the G line 294 of the signal line 290. The G pixel Wvalue calculation unit 820 supplies the calculated G pixel W value tothe phase difference detection pixel G value calculation unit 842.Calculation of the G pixel W value by the G pixel W value calculationunit 820 will be described with reference to FIGS. 19A and 19B.

The phase difference detection pixel G value calculation unit 842calculates the phase difference detection pixel G value based on the Gvalue of the G pixel (G pixel G value) supplied through the G line 294of the signal line 290 and the G pixel W value supplied from the G pixelW value calculation unit 820. The phase difference detection pixel Gvalue calculation unit 842 calculates the phase difference detectionpixel G value based on correlation between the W value and the G value.The phase difference detection pixel G value calculation unit 842supplies the calculated phase difference detection pixel G value to theimage synthesis unit 380.

Example of the Calculation of W Value of G Pixel by G Pixel W ValueCalculation Unit 820

FIGS. 19A and 19B are schematic diagrams showing an example of thecalculation of a W value of a G pixel by the G pixel W value calculationunit 820 according to the second embodiment of the present disclosure.

FIG. 19A schematically shows 9 pixels in a column in which the B pixel,the Gr pixel (in the same figure, referred to as a G pixel) and thephase difference detection pixel are arranged. FIG. 19A shows 9 pixelsadjacent to the G pixel (G pixel 550), the W value of which is to becalculated, in the column direction. The 9 pixels (the G pixels 510, 550and 590, the B pixels 530 and 570 and the phase difference detectionpixels 520, 540, 560 and 580) are pixels which hold the pixel valuesused when calculating the W value of the G pixel 550.

FIG. 19B shows an example of the calculation of the W value of the Gpixel by the G pixel W value calculation unit 820 along with the pixelvalues of the 9 pixels shown in FIG. 19A.

As shown in the inside of the 9 pixels arranged in one column of FIG.19B, when the G pixel W value calculation unit 820 calculates the Wvalue, the pixel value of the G pixel is configured by the G value andthe pixel value of the B pixel is configured by the B value. The pixelvalue of the phase difference detection pixel is configured by the Wvalue calculated by the phase difference detection pixel W valuecalculation unit 310. In FIG. 19B, the G values of the G pixels (Gpixels 510, 550 and 580) are denoted by G values 511, 551 and 591 andthe B values of the B pixels (B pixels 530 and 550) are denoted by the Bvalues 533 and 573. The W values of the phase difference detectionpixels (phase difference detection pixels 520, 540, 560 and 580) aredenoted by the W values 524, 544, 564 and 584.

Now, calculation of the W value of the G pixel 550 by the G pixel Wvalue calculation unit 820 will be described. In calculation of the Wvalue of the G pixel 550, the G pixel W value calculation unit 820calculates the W value based on correlation between the G value and theW value and the G value of the G pixel 550.

The W value (W_(G550)) of the G pixel 550 is calculated, for example,using Equations 17 to 19.W _(G550)=(W _(m) /G _(m))×G _(G550)  (17)W _(m) =a×W _(W520) +b×W _(W540) +c×W _(W560) +d×W _(W580)  (18)G _(m) =e×G _(G510) +f×G _(G550) +g×G _(G590)  (19)where, G_(G550) is the G value of the G pixel 550. In addition, G_(G510)is the G value of the G pixel 510 and G_(G590) is the G value of the Gpixel 590. In addition, W_(W520) is the W value of the phase differencedetection pixel (referred to as the W pixel) 520, W_(W540) is the Wvalue of W pixel 540, W_(W560) is the W value of the W pixel 560,W_(W580) is the W value of the W pixel 580.

In addition, a to g are weight coefficients and are equal to thosedescribed with reference to FIGS. 8A and 8B and thus description thereofwill be omitted.

In addition, W_(m) is a low frequency component of the W valuecalculated by applying a low pass filter using the position of the Gpixel 550 as a reference and, similarly, G_(m) is a low frequencycomponent of the G value calculated by applying a low pass filter usingthe position of the G pixel 550 as a reference.

That is, Equations 17 to 19 are equations for calculating the W valueusing correlation between the low frequency component of the G value andthe low frequency component of the W value, similarly to Equations 9 to11 shown in FIGS. 8A and 8B. As shown in Equation 17, the W value of theG pixel 550 is calculated based on correlation in low frequencycomponent between the G value and the W value and the G value of the Gpixel 550.

In FIG. 19B, the W value (W_(G550)) of the G pixel 550 corresponds tothe W value 554 (region in which a point is marked) shown in the G pixel550 shown on the right of an arrow. The pixel values used whencalculating the W value (W_(G550)) correspond to the pixel values (the Gvalues 511, 551 and 591 and the W values 524, 544, 564 and 584)surrounded by a thick line of the 9 pixels arranged in one column.

In the G pixel W value calculation unit 820, the G pixel W value iscalculated based on correlation in low frequency component between the Gvalue and the W value and the G value of the G pixel 550.

The phase difference detection pixel G value calculation unit 842calculates the phase difference detection pixel G value based oncorrelation in low frequency component between the G value and the Wvalue and the W value of the phase difference detection pixel, as shownin FIGS. 19A and 19B. In the phase difference detection pixel G valuecalculation unit 842, a low pass filter corresponding to the arrangementof pixels adjacent to the phase difference detection pixel to becalculated in the column direction is used.

Example of Operation of Imaging Apparatus

Next, the operation of the demosaicing unit 800 according to the secondembodiment of the present disclosure will be described with reference tothe figures.

FIG. 20 is a flowchart illustrating a processing procedure of an imagegeneration pixel W value calculation process (step S991) by the G pixelW value calculation unit 820 and the R-and-B pixel W value calculationunit 330 according to the second embodiment of the present disclosure.

This processing procedure is a modified example of FIG. 14 and isdifferent therefrom in that the W value of the G pixel is calculated bythe G pixel W value calculation unit 820. Since the other portions areequal to those of FIG. 14, the common parts with FIG. 14 are denoted bythe same reference numerals and description thereof will be partiallyomitted.

If it is determined that the pixel to be determined is the G pixel (stepS953), by the G pixel W value calculation unit 820, the W value of the Gpixel is calculated using correlation between the W value and the Gvalue (step S992) and then the process progresses to step S956.

FIG. 21 is a flowchart illustrating a processing procedure of a G valuecalculation process (step S995) by the R-and-B pixel G value calculationunit 341 and the phase difference detection pixel G value calculationunit 842 according to the second embodiment of the present disclosure.This processing procedure is a modified example of FIG. 15 and isdifferent therefrom in that the G value of the phase differencedetection pixel is calculated by the phase difference detection pixel Gvalue calculation unit 842.

Since the other portions are equal to those of FIG. 15, the common partswith FIG. 15 are denoted by the same reference numerals and descriptionthereof will be partially omitted.

If it is determined that the pixel to be determined is not the G pixel(step S962), the G value is calculated using correlation between the Wvalue and the G value (step S992) and then the process progresses tostep S966. The G value of the R pixel or the B pixel is calculated, forexample, using Equations 9 to 14 shown in FIGS. 8A and 8B. The G valueof the phase difference detection pixel is calculated, for example,using the G value and the W value of the image generation pixel adjacentto the phase difference detection pixel, the G value of which is to becalculated, in the column direction, similarly to Equations 17 to 19 ofFIG. 19.

According to the second embodiment of the present disclosure, it ispossible to improve image quality of image data, similarly to the firstembodiment of the present disclosure, even when the G pixel W value iscalculated based on correlation in low frequency component between the Gvalue and the W value and the G value of the G pixel 550.

3. Modified Example

In the first and second embodiments of the present disclosure, it isassumed that, in the arrangement of pixels of the imaging device, asshown in FIG. 4, the row in which the image generation pixels arearranged and the row in which the phase difference detection pixels arearranged are alternately arranged. However, the present disclosure isnot limited thereto and is applicable to the case of using the imagingdevice having other pixel arrangements, similarly to the first andsecond embodiments of the present disclosure. The present disclosure iseven applicable to the case where a color filter pixel is included inthe phase difference detection pixel, similarly to the first and secondembodiments of the present disclosure.

In FIG. 22, an example of an imaging device in which image generationpixels arranged in two columns and phase difference detection pixelsarranged in two columns are alternately arranged will be described. InFIG. 23, an example of an imaging device in which phase differencedetection pixels each including a color filter (G filter) transmittinggreen (G) light will be described.

Example of Arrangement of Pixels in Imaging Device

FIG. 22 is a schematic diagram showing an example of arrangement ofpixels of an imaging device in which image generation pixels arranged intwo columns and phase difference detection pixels arranged in twocolumns are alternately arranged as a modified example of the firstembodiment of the present disclosure.

FIG. 22 corresponds to FIG. 4 showing arrangement of pixels of theimaging device 210 according to the first embodiment of the presentdisclosure. That is, FIG. 22 shows a region (region 860) of some pixels(16×16 pixels) among pixels configuring the imaging device according toa third embodiment of the present disclosure. As shown in the region860, in the modified example of the first embodiment of the presentdisclosure, two columns in which the image generation pixels arearranged and the two columns in which the phase difference detectionpixels are arranged are alternately arranged. That is, as shown in FIG.22, in the x-axis direction, the image generation pixel, the imagegeneration pixel, the phase difference detection pixel, the phasedifference detection pixel, the image generation pixel, the imagegeneration pixel, the phase difference detection pixel, the phasedifference detection pixel, . . . are alternately arranged.

In such pixel arrangement, the R-and-B pixel G value calculation unit341 calculates the G value using the W value and the G value of thepixel adjacent to the pixel, the G value of which is to be calculated,in the column direction (for example, calculation shown in FIG. 19).Thus, it is possible to perform the demosaicing process, similarly tothe first embodiment of the present disclosure.

FIG. 23 is a schematic diagram showing an example of arrangement ofpixels of an imaging device in which phase difference detection pixelseach including a G filter are arranged as a modified example of thefirst embodiment of the present disclosure.

FIG. 23 corresponds to FIG. 4 showing arrangement of pixels of theimaging device 210 according to the first embodiment of the presentdisclosure, similarly to FIG. 22.

In a region 870 shown in FIG. 23, phase difference detection pixels(phase difference detection pixels 875 to 878) each including the Gfilter is shown.

In such pixel arrangement, the G value of the phase difference detectionpixel is calculated by calculating the pixel value corresponding to thelight shielding region and summing the calculated pixel value and thepixel value of the phase difference detection pixel, similarly toEquations 1 to 8 shown in FIGS. 6A to 6F. Since the G value iscalculated instead of the W value, division of “2” of Equations 2, 4, 6and 8 becomes unnecessary.

The G values of the R pixel and the B pixel are calculated from thehistogram based on the distance information. The R value and the B valueare calculated from the histogram based on the distance information,similarly to the first embodiment of the present disclosure.Accordingly, it is possible to perform the demosaicing process,similarly to the first embodiment of the present disclosure.

In the embodiments of the present disclosure, by performing thedemosaicing process using the pixel value of the phase differencedetection pixel, it is possible to improve image quality of image data.That is, in the embodiments of the present disclosure, by calculatingthe W value from the pixel value of the phase difference detection pixeland performing the demosaicing process using the W value, it is possibleto increase the number of pixel values which may be referred to upon thedemosaicing process and improve resolution or improve interpolationprecision. By calculating the distance information from the pixel valueof the phase difference detection pixel and performing the demosaicingprocess using the distance information so as to perform interpolationwhile identifying objects having the same color and having differentdistances, it is possible to improve interpolation precision.

In general, the image generation pixel (W pixel) including the W filteris apt to be saturated. Accordingly, the imaging apparatus including theW pixel includes an exposure control circuit for the W pixel. In theembodiments of the present disclosure, by estimating the W value fromthe pixel value of the phase difference detection pixel in which aportion of the pupil divided light is shielded, saturation hardly occurseven when the exposure control circuit is not included (up to twice asaturated amount of a light receiving surface of a light receivingelement may be accumulated). By estimating the W value from the pixelvalue of the phase difference detection pixel and using the W value inthe demosaicing process, it is possible to improve a signal/noise ratio(S/N) ratio.

Although the case where the color filters included in the imagegeneration pixels are color filters having three primary colors (RGB) isdescribed, the present disclosure is not limited thereto. For example,the present disclosure is equally applicable to the case where colorfilters having complementary colors are included in the image generationpixels.

Although the case where the phase difference detection pixel receivesportions of light pupil divided into two is described, the presentdisclosure is not limited thereto. For example, the embodiments of thepresent disclosure is applicable to the case where two light receivingelements are included and the phase difference detection pixels capableof receiving the pupil divided light by the light receiving elements,thereby improving the image quality of image data.

Although the imaging apparatus is described as an example in theembodiments of the present disclosure, the present disclosure is notlimited thereto. For example, the process according to the firstembodiment of the present disclosure may be performed by an externalapparatus (for example, a personal computer) of the imaging apparatusbased on recorded RWA data.

The embodiments of the present disclosure are examples for embodying thepresent disclosure. As described in the embodiments of the presentdisclosure, the matter of the embodiments of the present disclosurecorresponds to the specific matter of the claims. Similarly, thespecific matter of the claims corresponds to the matter of theembodiments of the present disclosure having the same names. The presentdisclosure is not limited to the embodiments and various modificationsof the embodiments may be implemented without departing from the scopeof the present disclosure.

The processing procedures described in the embodiments of the presentdisclosure may be a method having a series of procedures, a program forexecuting the series of procedures or a recording medium for storing theprogram. As the recording medium, for example, a compact disc (CD), amini disc (MD), a digital versatile disc (DVD), a memory card, a Blu-rayDisc (registered trademark), or the like may be used.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2010-282524 filed in theJapan Patent Office on Dec. 20, 2010, the entire contents of which arehereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing apparatus comprising: anestimation unit configured to estimate an image generation pixel valuecorresponding to a position of a phase difference detection pixel of animage data, based on a determination pixel value of the image data,wherein the image data comprises the phase difference detection pixeland an image generation pixel, wherein the phase difference detectionpixel generates the determination pixel value for making a focusdetermination and the image generation pixel generates the imagegeneration pixel value for generating an image; and an interpolationunit configured to interpolate image generation pixel values of pixelsconfiguring the image data based on the estimated image generation pixelvalue and the generated image generation pixel value.
 2. The imageprocessing apparatus according to claim 1, further comprising a distanceinformation generation unit configured to generate distance informationbased on a defocus quantity calculated based on the determination pixelvalue for each phase difference detection pixel, wherein theinterpolation unit interpolates the image generation pixel values ofpixels configuring the image data based on the estimated imagegeneration pixel value, the distance information and the imagegeneration pixel value generated by the image generation pixel.
 3. Theimage processing apparatus according to claim 2, wherein theinterpolation unit sets a color to be an interpolation of a referencepixel as a target color using a pixel to be interpolated as thereference pixel, calculates the distance information of the imagegeneration pixel based on distance information of the phase differencedetection pixel if the image generation pixel value of the referencepixel is interpolated using the distance information, sets a pixel whichholds an image generation pixel value related to the target color amongpixels located in a predetermined range from the reference pixel as atarget pixel, detects the target pixel within the predetermined rangefrom the reference pixel in a distance specified by the distanceinformation of the target pixel based on the distance information, andinterpolates the image generation pixel values by setting an averagevalue of the image generation pixel value related to the target color ofthe detected target pixel to the image generation pixel value related tothe target color of the pixel to be interpolated.
 4. The imageprocessing apparatus according to claim 3, wherein the interpolationunit creates a frequency distribution of pixels for the distancespecified by the distance information of the target pixel, sets adistance, in which a frequency of the frequency distribution belongs toa maximum class, as a reference, detects the target pixel correspondingto the distance within the predetermined range from the reference, andsets the average value of the image generation pixel value related tothe target color of the detected target pixel as the image generationpixel value related to the target color of the pixel to be interpolated.5. The image processing apparatus according to claim 1, wherein thephase difference detection pixel comprises a microlens, a lightreceiving element and a light shielding unit disposed between themicrolens and the light receiving element, wherein the microlens isconfigured to focus subject light, the light receiving element isconfigured to generate an image generation pixel value by receiving thesubject light and the light shielding unit is configured to partiallyshield the subject light, and the estimation unit calculates the pixelvalue related to the partially shielded light based on the determinationpixel value generated by the phase difference detection pixel to beestimated and the determination pixel value generated by the phasedifference detection pixel adjacent to the phase difference detectionpixel to be estimated, and estimates the image generation pixel value ofthe position of the phase difference detection pixel based on thecalculated pixel value and the determination pixel value of the phasedifference detection pixel to be estimated.
 6. The image processingapparatus according to claim 5, wherein the image generation pixelincludes a red pixel covered by a red filter for shielding light otherthan a wavelength region indicating red, a blue pixel covered by a bluefilter for shielding light other than a wavelength region indicatingblue, and a green pixel covered by a green filter for shielding lightother than a wavelength region indicating green, the phase differencedetection pixel is covered by a white filter or a transparent layertransmitting light of a visible light region, and the estimation unitestimates the image generation pixel value related to white as the imagegeneration pixel value of the position of the phase difference detectionpixel.
 7. The image processing apparatus according to claim 6, whereinthe interpolation unit sets a color to be an interpolation of areference pixel to a target color using a pixel to be interpolated asthe reference pixel, interpolates the image generation pixel valuerelated to white of the image generation pixel values of pixelsconfiguring the image data, and interpolates the image generation pixelvalues related to the target color of the reference pixel, based on theimage generation pixel value related to the target color of the pixelslocated within a predetermined range from the reference pixel, the imagegeneration pixel value related to white of the pixels covered by thesame filter as the pixel to be interpolated as the pixels located withinthe predetermined range from the reference pixel and the imagegeneration pixel value related to the white of the reference pixel. 8.The image processing apparatus according to claim 7, wherein theinterpolation unit interpolates the image generation pixel valuesrelated to the target color of the reference pixel, based on a lowfrequency component of the image generation pixel value related to thewhite calculated based on the image generation pixel value related tothe white of the pixels located within the predetermined range from thereference pixel, a low frequency component of the image generation pixelvalue related to the target color calculated based on the imagegeneration pixel value related to the target color of the pixels coveredby the filter of the target color located within the predetermined rangefrom the reference pixel, and the image generation pixel value relatedto the white of the reference pixel.
 9. The image processing apparatusaccording to claim 7, wherein, in the imaging device, a first lineconfigured by arranging the image generation pixels in a specificdirection and a second line configured by arranging the phase differencedetection pixels in the specific direction are alternately arranged inan orthogonal direction orthogonal to the specific direction, and theinterpolation unit interpolates the image generation pixel valuesrelated to the white of the green pixel, based on a low frequencycomponent of the image generation pixel value related to the whitecalculated from the image generation pixel value related to the white ofthe phase difference detection pixels located within the predeterminedrange in the orthogonal direction of the reference pixel and a lowfrequency component of the image generation pixel value related to thewhite calculated from the image generation pixel value related to greenof the green pixels located within the predetermined range, and theimage generation pixel value related to green of the reference pixel.10. An image processing apparatus comprising: a distance informationgeneration unit configured to generate distance information based on adefocus quantity calculated for each phase difference detection pixelbased on a determination pixel value of image data, wherein the imagedata comprises a phase difference detection pixel and an imagegeneration pixel, wherein the phase difference detection pixel generatesthe determination pixel value for making a focus determination and theimage generation pixel generates the image generation pixel value forgenerating an image; and an interpolation unit configured to interpolatethe image generation pixel value of a pixel to be interpolated amongpixels configuring the image data based on the generated distanceinformation and the generated image generation pixel value.
 11. An imageprocessing method comprising: estimating an image generation pixel valuecorresponding to a position of a phase difference detection pixel of animage data, based on a determination pixel value of the image data,wherein the image data comprises the phase difference detection pixeland an image generation pixel, wherein the phase difference detectionpixel generates the determination pixel value for making a focusdetermination and the image generation pixel generates the imagegeneration pixel value for generating an image; and interpolating imagegeneration pixel values of pixels configuring the image data based onthe estimated image generation pixel value and the generated imagegeneration pixel value.
 12. The image processing method according toclaim 11, comprising correcting the image data based on a predeterminedsignal processing technique.
 13. The image processing method accordingto claim 12, wherein the predetermined signal processing techniquecomprises one or more of: a black level correction, a defect correction,a shading correction, and/or a mixed color correction.
 14. The imageprocessing method according to claim 11, comprising detecting deviationof focus of a pair of phase difference detection pixels.
 15. The imageprocessing method according to claim 14, comprising demosaicing based onthe image data generated by the phase difference detection pixel, theimage data generated by the image generation pixels, and the detecteddeviation of focus.
 16. The image processing method according to claim11, comprising correcting color balance of the image data based on apredetermined reference color.
 17. The image processing method accordingto claim 16, comprising performing a non-linear conversion on thecorrected image data.
 18. A non-transitory computer-readable storagemedium having stored thereon, a computer program having at least onecode section for image processing, the at least one code section beingexecutable by a computer for causing the computer to perform stepscomprising: estimating an image generation pixel value corresponding toa position of a phase difference detection pixel of an image data, basedon a determination pixel value of the-image data, wherein the image datacomprises the phase difference detection pixel and an image generationpixel, wherein the phase difference detection pixel generates thedetermination pixel value for making a focus determination and the imagegeneration pixel generates the image generation pixel value forgenerating an image; and interpolating image generation pixel values ofpixels configuring the image data based on the estimated imagegeneration pixel value and the generated image generation pixel value.19. The non-transitory computer-readable storage medium according toclaim 18, comprising correcting color balance of the image data based ona predetermined reference color.
 20. The non-transitorycomputer-readable storage medium according to claim 19, comprisingperforming a non-linear conversion on the corrected color balance of theimage data.